Dennis Cokely shares the RID/NAD written response to his initial letter requesting clarification on the “enhanced” NIC test with annotations on the report provided by the governing entities.
I want to thank StreetLeverage for creating a forum where issues affecting sign language interpreters and the field of sign language interpreting can be raised and discussed thoughtfully and respectfully. This forum has allowed me the opportunity to share my communications with the RID Board on questions I believe need to be answered regarding the “enhanced” NIC certification test. My last posting on this topic resulted in a rich and stimulating discussion.
The questions raised in my post, Defenders of Certification: Sign Language Interpreters Question “Enhanced” RID NIC Test, and that will be raised here, are not raised out of nostalgic ties; not raised out of a desire to cling to the past; not raised out of a desire to foment dissent. In truth, I raise these questions to provide certified sign language interpreters (and, indeed, all members of RID) with the information and facts necessary for us to serve as defenders, not critics, of the current NIC evaluation. Sadly I believe that, to date, we have not been provided with such information or facts.
What follows is a letter I sent to the RID Board on June 25, 2012 in response to the “comprehensive report” that was released by the Board on June 7, 2012, which was in response to my March 18, 2012 letter questioning aspects of the “enhanced” NIC test.
My Intent
I want to explicitly state my intent in writing these letters and raising these questions is not to spur divisiveness but rather to garner transparency and to seek the underlying rationale for decisions that were made without full involvement of the certified membership. To date I have not received a response from the Board to this letter, although I have received an acknowledgement that my email letter has been received.
I truly hope that a meaningful response to the issues I have raised will be provided in a timely and public manner. If such a response is forthcoming, I would suggest that the approach of “It is because we say it is” is decidedly not a useful strategy in addressing an issue as important to RID certified members as this is; the response must be one that comports well with the more than forty years of experience RID has assessing interpreting competency and that comports with the real world experiences of practitioners.
My hope is that my postings on this issue can, and will, spur discussions that can better inform our decision-making process and, ultimately, improve our assessment of who is qualified to become “one of us”.
My Letter
June 25, 2012
To Presidents Prudhom and Scoggins and members of the RID Board:
Thank you for your letter dated June 7 and for the “comprehensive report” that was attached. Following you will find my response to the report and some additional issues/questions I have that were raised by the report.
You are correct that my letter was made public. But, please know that it was made public only after more than a month had passed after I had sent the letter to all members of the Board during which time its receipt was not even acknowledged by the Board. While I appreciate the demands on a volunteer Board, I would suggest that the serious questions I raised deserved a much more timely response (and at least the courtesy of an acknowledgment of receipt by the Board) and the answers which I requested should not have taken three months to assemble. I also believe that the “community discussion” its release created is definitely a positive discussion and one that should have occurred as the “enhancements” were conceived and most definitely before they were implemented. I hope that continued, open and public discussion will only help to strengthen our decisions and more fully engage the RID membership in these critical decisions and programs. I must tell you that I do intend to post this letter to you in a public forum in a couple of weeks – I firmly believe that providing a forum for members discuss these issues not only allows a fuller airing of issues, but also allows the Board to have a more accurate picture of the pulse of the membership than it appears to have on this issue.
I am sure you can appreciate the fact that the report that was generated in response to my letter is, based on the numerous emails I have received and numerous postings on social media, viewed by many as quite a defensive, reactive and inadequate response. My intent here, and throughout, is not to create an atmosphere of confrontation or to incite divisiveness. Rather, my intent is that, unlike decisions taken in the past, we can all agree that the decisions we make in the certification program are based on empirical data, not the feelings or beliefs of a small group that may not well-represent the membership. In your letter to me you state “NAD and RID fully support the work and direction of the enhanced NIC…”. I humbly and respectfully strongly disagree. At the very least in the case of RID, I think it is much more accurate and truthful to state that the RID Board fully supports the “enhanced” NIC. I say this because there is considerable discontent and unrest among the membership regarding the “enhanced” NIC. I also believe there has been anything but acceptable and appropriate levels of transparency regarding decisions surrounding these “enhancements” which is quite surprising for what the Board claims is a “membership driven” organization. I think that it is fair to say that for many members there is a clear sense that critical decisions that define who we are as interpreters have been made without significant and meaningful involvement of, and engagement with, the members and most definitely without meaningful discussion at our regional and national conferences, again belying the notion of RID as a “membership driven” organization. I certainly urge you to make good on your pledge to “…dedicated communication effort…now and in the future” and hope you do so proactively rather than reactively.
The “comprehensive report” provided many statements that asserted that the “enhanced” NIC was valid and reliable but provided no empirical or psychometric data to support such assertions. Unfortunately I see no proactive involvement of the membership in many of the issues I have raised; I see no checks and balances built into the current “enhanced” NIC that can ensure impartiality and objectivity; I see no independent checks and balances of the design and implement work off the consulting group; I see no concern for the negative reactions and responses of members that this “enhanced” NIC is not based on sound empirical data; and unfortunately I see no evidence in the Board’s actions that can support its claim that RID, in this arena at least, is a “membership-driven” organization.
I fully appreciate the pressures and demands on an all-volunteer Board having been there myself. I trust you will accept my response and the questions I raise below in the spirit of moving toward a stronger and more cohesive RID. However, as I stated before, the fact is that there is growing discontent among the membership with the manner in which decisions are taken within the organization and a growing feeling of complete disenfranchisement within the organization. There already is a growing number of members and an increasing number of states in the US who believe that RID certification has become meaningless and irrelevant. More importantly, there is an increasing chorus of RID members crying for an alternate organization and that chorus grows as members take the “enhanced” NIC. I, for one, am saddened to see that number grow. But if sound, logical answers to the questions I have previously raised and also raise below cannot be provided then, unfortunately, I would have no option but to join their number and advocate loudly and passionately for creation of an alternate organization. I would be incredibly saddened should that come to pass.
I thank you for your willingness to engage in this discussion and I am willing to continue this very important discussion in any forum you feel will be beneficial (email, VideoPhone, phone, face-to-face meetings, etc.).
I eagerly await your response.
Sincerely,
dennis
Dennis Cokely, Professor
Director, American Sign Language Program
Director, World Languages Center
Chair, Department of Languages, Literatures and Cultures
Northeastern University
Boston, MA 02115
My Response to the June 7, 2012 “Comprehensive Report” on the NIC “enhancements
1) Before raising specific questions generated by the report, I ask that the Board offer an explanation to the membership for why the same consultant firm that was directly responsible for the design and implementation of the “enhanced” NIC was also asked to write the interim report which asserts that the “enhanced” NIC is valid and reliable. While I, in no way, intend to cast aspersions on the Caviart group, I believe it is incumbent upon the Board explain why this is not a clear conflict of interest and why an independent psychometrician was not engaged to review the overall process and write the report? Given the catastrophic recent NIC history and the absolute appearance of a possible conflict of interest, I would urge the Board to address this as quickly as possible. Why should we have faith in the validity of the Caviart report? Why is this not a conflict of interest given that Caviart has a vested interest in convincing the Board and members that its initial test development is valid and reliable?
2) It comes as quite a surprise to me and I am sure to most RID members that, on page #2 of the report, we learn that the profile of a “certified NIC (level 1) interpreter” was developed by 14 members of the NIC Task Force, an NIC Scoring Group (whose composition is unknown) and an unidentified group of “subject-matter experts”. This means that, if one includes the Board (which, I assume, approved the profile), it would appear that fewer than 100 members had a hand in determining this profile. This represents less than .009% of the membership. Why was this profile not circulated to the membership for comment and input? How is not circulating this profile consistent with claims of a “membership driven” organization?
3) It is also troubling is that the profile of a “certified NIC (level 1) interpreter” is offered without any rational, explanation or justification. What empirical basis is there to support this profile? What data is there to suggest that a “…vocabulary at a first year undergraduate level…” is appropriate? What is meant by “…quickly adapt to different situations and communication styles”?
4) But what is even more troubling on page #2 is the incredibly dismissive tone in the report intended for those who might question or challenge (“That is fine.”). While it is certainly true that “…no profile will satisfy everyone”, I believe that the leadership owes it to the members to create an environment in which questions and challenges are welcomed because, presumably, the leadership has data to support its choices and decisions and thus can appropriately respond to such questions and challenges. The tone here smacks of “it is because we say it is”. If this tone emanates from the consultant group then perhaps we have retained the wrong group. If the tone emanates from the leadership, then the situation is even worse that I thought and we definitely have a crisis of leadership. How is such a tone consistent with a “membership driven” organization? Does the membership not, at least, deserve the courtesy of opportunities to discuss the definition of the threshold that marks who is “one of us”?
5) Also, on page #2, it appears that a decision has been made that there will be “levels” to the NIC (“currently called Level 1”). If this is indeed the case, this represents a significant departure from past practice (the recently aborted iteration of the NIC notwithstanding) in which we have had a “generalist then specialist” approach to certification. I ask that the Board release the overall comprehensive master plan mentioned on page #3 for the certification program as well as the specific criteria for determining “…higher and specialized levels”. What is the overall master plan for the certification program?
6) Again on page #2, I believe it is totally inaccurate to make the claim that “…this statement summarizes what both organizations believe…”. At the present time I believe that it is accurate to say, in the case of RID at least, that this statement summarizes what the Board, the NIC Task Force and an NIC Scoring Group whose composition is unknown believes. The membership were not at all engaged in this discussion nor invited to provide feedback on this issue. Can the RID Board explain how it purports to represent the membership on this issue when it has not fully engaged the membership?
7) On page #3, it is extremely troublesome to read that the claim “…more interpreter/consumer interactions occurred remotely…” is based on the feelings of the NIC Task Force rather than on empirical data. In fact, according to a recently completed survey of RID Certified and Associate members conducted by the National Interpreter Education Center, 70% of interpreters who responded report they do absolutely no work in VRS and 92% do no work in VRI. These data make it clear that it is still the case that from interpreters’ experience, the overwhelming majority of the work that interpreters do is conducted in face-to-face interactions. While it may be that Deaf people are using more VRS/VRI, at this point in our history the fact is that the vast majority of interpreters do not work in those settings. This begs the question of why we developed an assessment approach that appears predicated on beliefs, settings, assumptions in which only 30% and 8% of us work?
8 ) Given the fact that some members of the NIC Task force have VRS/VRI ties, one also has to wonder again about the wisdom of relying on feelings rather than empirical data. Why are feelings used over empirical data? Again I do not intend to cast aspersions, but when segments on the “enhanced” NIC have durations of three to five minutes (which resemble durations found in VRS calls), one wonders what response the Board might have to the fact that there is the appearance of a conflict of interest for members of the NIC Task force who do have VRS/VRI ties?
9) On page 5 the process by which the “content and format” of the “enhanced” NIC is described. The logic of this process seems, to me, to be somewhat circular and rather flawed. If I understand the process correctly, raters used the previous NIC exam, material with which they were intimately familiar having viewed the material dozens and dozens of times. They were then asked to identify portions of that material that they believed “…most effectively discriminated the skills of interpreters at the level described”. From a linguistic and sociolinguistic perspective, what is critical is that they selected these portions from within a larger context and those portions were taken from within a larger sample of a candidate’s work. Thus raters had significant background information influencing the choice of segments that they believed “…most effectively discriminated the skills of interpreters at the level described”. Scoring criteria were then identified using a “proven algorithm”. While the algorithm may be “proven”, the elements (and the empirical support for those elements) to which the algorithm was applied need to be known. This is especially true since the previous iteration of the NIC listed rating elements that conflated language elements (e.g. articulation, use of space) with interpretation elements. What are the elements to which raters will apply the scoring algorithm and what is the empirical basis for those elements?
10) Having demonstrated that portions from within a larger context and taken from within a larger sample of a candidate’s work could be reliably rated, how then is it reasonable to conclude that portions generated with no larger context and from no larger work sample could or would suffice as a valid indicant of a candidate’s competence?
11) Having the Scoring Group finalize the profile, develop the scoring criteria, rate previously rated samples and then discuss their holistic ratings, places enormous and unchecked power in the hands of a small group. Why was there no independent confirmation of the Scoring Group’s work? Why were not different groups engaged in segments of this undertaking? How large was the Scoring Group? Who were its members?
12) On page 6 we are told that the final vignettes were “…believed, by the Scoring Group, to have the most appropriate content, vocabulary and stimulus material”. The Scoring Group then developed new scoring criteria – again enormous and unchecked power in the hands of a small group. Is there a rationale for this? Are we to believe that there is a different set of criteria for each vignette? If so, what is the full set of criteria used to rate a candidate and what is the empirical basis for each criterion? Why should we believe the Scoring Group is qualified to make such determinations?
13) I submit that the assertion that “While content validity is critical, face validity is not” critical completely ignores the recent past with regard to the NIC and reflects a complete lack of understanding of our assessment and institutional history. Certainly face validity is unquestionably important for market acceptance. And most certainly one incredibly important segment of that market is the RID Certified and Associate membership and students in Interpreter Education Programs. If certification is to have any value, these stakeholders simply must feel and believe that the high-stakes assessment that is the NIC, at least looks like what interpreters do regularly. While it may be true that an assessment of interpretation skills does not have to look like what interpreters do regularly, one would think that, given the last significantly flawed iteration of the NIC, the Board would most definitely want the assessment to look like what interpreters do regularly. Given the incredibly negative issues surrounding the last iteration of the NIC, why would the Board, the NIC Task Force and the NIC Scoring Group endorse an assessment approach that most clearly lacks face validity and seems not to be widely supported?
14) On page 8 the dismissive tone of this report continues with the assertion that there is “…absolutely no merit to this suggestion.” In the present climate, I assert precisely the opposite – if the members of RID do not believe that the assessment is valid and looks valid, if we cannot defend it, if we cannot/will not encourage those who are not yet certified to seek certification, then the assessment process is seriouslyflawed. I believe that there is incredible merit to the need for face validity of the NIC. The report asserts that the “enhanced” NIC “…has significant face validity…”, but this is just another example of “it is because we say it is”. No empirical data is offered here. However, candidates who have taken the “enhanced” NIC and who have contacted me almost unanimously say that they think that the “enhanced” NIC did not fairly sample their interpreting skills, did not look like what they do on a regular basis and did not allow them to demonstrate what they do when they interpret. Can the Board, the NIC Task Force or the NIC Scoring Group provide empirical data that candidates do indeed feel the “enhanced” NIC fairly samples and assesses their work? What percent of those who have taken the “enhanced” NIC report that it “fairly sampled” their interpreting competence? I doubt such data even exists or is collected.
15) On page 8 the report states that panels were asked to “…identify the amount of time that it takes to accurately assess a candidate’s skill.” Again I ask whether there is any empirical data to support this approach; what we have is self-report data, drawn from an individual’s various experiences that are based on samples that are highly contextualized. If I state that in a given real-world context I can determine in two or three minutes whether I can accurately assess skill for this particular situation, I cannot validly apply that determination to other contexts. Why have we have taken timing information based on contextualized self-reports and applied that timing information to de-contextualized vignettes? Is there any empirical support for doing so? Are we asking raters trained to assess the former NIC to make this determination? If so, by what justification? Or is this another instance in which the ubiquitous Scoring Group
16) The unidentified subject matter “experts” (number and qualifications unknown) believe that “niceties” can be excluded because they “…provide little information about a candidate’s skill”. It certainly is true that, from an interpreter’s perspective, the beginnings of interactions are often not challenging and thus may not be fully representative of a candidate’s skill set. However, there is clear linguistic empirical data to show that these “niceties” are often essential to an interpreter’s overall ease, comfort and comprehension of a speaker/signer and thus important to rendering successful interpretations. Indeed, one cannot “…go right to the heart of the communication encounter.” Linguists and Sociolinguists have shown clearly that successful communication is an evolved and negotiated interaction; one cannot properly and fully understand “the heart” of an interaction while ignoring or not having access to the “skeleton” and the “flesh” that surround the “heart”. We return again to the face validity question. How is it that we can warrant that those who pass the NIC can “…relay the essence of a message…” when our assessment strips away all of the context, linguistic background and interactional unfolding that leads up to “…the essence of a message…”? What is the empirical data to support this decision? Certainly the “comprehensive report” does not address this question.
17) The use of “tower crane operators” is an insulting and ignorant analogy at best (no offense intended to tower crane operators). Granted there is considerable pressure in being a crane operator. However, there is also a clear “right and wrong” result, the ball is manipulated correctly or it is not; the result is black or white. The results on a crane operator test are plain for all to see – the wall comes down or it does not (true some might be more efficient than others; but ultimately if the wall does not come down the operator has not been successful). However, as any interpreter knows, interpreting is anything but a “black or white” cognitive task. There are a finite number of moves that are possible with a tower crane. However, as any interpreter knows there are myriad possibilities for rendering successful interpretations in any interaction because we are dealing with people not machines. And while the obstacles through which crane operators must move to demonstrate their skill are finite, interpreters clearly know that because communication involves a range of human beings, a range of situations, and a range of communicative goals, the obstacles through which we must maneuver are virtually infinite. It does not follow logically that because tower crane operators can be assessed by an examination that lacks face validity that interpreters should accept, much less endorse, any examination that lacks face validity. Why should we accept a lack of face validity?
18) The use of “police officers” is also not an appropriate analogy for our purposes. Police officers spend months and months training at the police academy; they do not get to take their performance exam until/unless they have performed well in their training routines during which they have fired their guns countless times. The police officer test itself is valid because it presumes months and months of training that is specific to being a police officer. However, in our case, the claim that NIC “…standards include requirements for education…”(pg. 6), make using police officer testing a false analogy. Our forthcoming “requirement for education” is, at this point, only a BA/BS degree but not a degree in interpreting. So, even though the police officer test may not be indicative of what they do on a daily basis, police cadets have had months of training and practice to demonstrate what they will do on a daily basis. RID has no such warrantied, specific interpreter educational background requirement for NIC candidates. Lacking the focused, specific training/education that is required of police officers (and tower crane operators) and that forms the foundation on which their tests can rest, why would we adopt an assessment approach that presumes such focused, specified training/education when we do not yet have such focused, specified training/education?
19) On page 9 the report asserts that a testing program is fair if “…every candidate has the same opportunity to demonstrate his or her competence.” Again, based on the numerous comments I have received from those who have taken the “enhanced” NIC, the majority feel they did not have an opportunity to demonstrate their competence. All candidates may be presented with the same logistically structured opportunity with the NIC, but if that opportunity is believed by a majority of candidates not to afford them an “…opportunity to demonstrate his or her competence” or is believed by them to be a flawed opportunity, how can such a program be judged as fair? How does the Board respond to candidates who feel they did not have an opportunity to demonstrate their competence?
20) Page 9 states that there is “…no evidence…that suggests that physical endurance is required for the job.” This patently ignores significant research in signed and spoken language interpreting research on this matter. It patently fails to realize that physical endurance is not the critical issue in interpretation — cognitive endurance definitely is!!!! This is extremely well documented in the literature. With the “enhanced” NIC vignettes ranging from only three minutes to five minutes in length (although one LTA emailed me saying that they carefully timed each scenario and the actual range was between 1:20 and 3:20!!), it is virtually impossible to see how we assess cognitive endurance. This is, after all, one of the most important reasons why we advocate for the use of teams in many situations. If there is no evidence in the “Role Delineation Study” that speaks to cognitive fatigue, then I suggest that that study is seriously flawed. If there is such evidence in the Study and the “enhanced NIC” ignores this, then the program is seriously flawed. Why do we think that cognitive fatigue is not a critical factor to assess? And why was the “Role Delineation Study” not more widely vetted and shared?
21) Page 10 states that “…RID has carefully specified the testing conditions…”. Based on information I have received from candidates who have taken the “enhanced” RID, this means that candidates must be seated and must remain so for the duration of the test. As any interpreter knows, when an interpreter is seated, his/her range of motion is severely restricted and thus his/her use of space for semantic/linguistic purposes is also restricted. Given that we have never restricted candidates in this manner, what empirical evidence is there that placing interpreters in such restricted conditions will produce samples of their work that are indicative of their overall competence? Why would we want to restrict/constrain the use of semantic/linguistic space?
22) Page 11 proclaims the hope that the “enhanced” NIC will “…earn the value and respect from consumers that it deserves”. I submit that the “enhanced” NIC cannot earn respect from consumers until and unless it is accepted, embraced and valued by practitioners. This status report references a number of reports and studies that, to my knowledge, have not been made available to the RID membership. When can members expect release of all reports that are referenced in the Status Report?
My Previous Questions
In my initial letter to the Board, I asked nine questions. I was told that a “comprehensive report” would be issued that would address these questions. Unfortunately, I do not believe that any of the questions has been answered satisfactorily.
1) RID members need a more thorough explanation of why time and a simple mathematical formula should be the primary drivers behind the format of the certification test; if this is not true, then a clear explanation should be provided for how the current 4-minute per vignette test segmentation was determined.
Answer provided in the “comprehensive report” is inadequate
2) An explanation for the process/principles used in the selection of and/or development of the vignettes be made known to the membership.
Answer provided in the “comprehensive report” is inadequate
3) A full explanation of the empirical justification for this 4-minute approach must be provided to the membership.
Answer provided in the “comprehensive report” is inadequate
4) A clear explanation of the rationale and justification for placing candidates at such an interpreting disadvantage must be provided to the membership.
Answer provided in the “comprehensive report” is inadequate
5) A detailed explanation of the rational for, and empirical support for this decision and this deviation from forty years of experience is also needed by the membership.
Answer provided in the “comprehensive report” is inadequate
6) If there is evidence that supports the claim that a 4-miute sample can validly and reliably assess a candidate’s ability to assess sustained quality over time, then it must be made known to the membership.
Answer provided in the “comprehensive report” is inadequate
7) What are the various English and ASL grammatical and semantic features in vignettes that raters will be assessing and do these five 4-minute vignettes provide sufficient linguistic and discourse variation to elicit an appropriate range of English and ASL grammatical and semantic features?
Answer provided in the “comprehensive report” is inadequate
8 ) Since using discretion in selecting assignments is one of the core operating principles of our long-standing Code, the rationale for adopting an “all or nothing” approach must be made clear to the membership.
Answer provided in the “comprehensive report” is inadequate
9) A clear, empirically supported explanation of why the current NIC assessment is valid and can be reliably assessed by raters must be provided to the membership.
Answer provided in the “comprehensive report” is inadequate
