Posted on 33 Comments

Sign Language Interpreters Seek Clarity to Defend RID NIC Certification

Sign Language Interpreter Seeking Clarity

Dennis Cokely shares the RID/NAD written response to his initial letter requesting clarification on the “enhanced” NIC test with annotations on the report provided by the governing entities.

I want to thank StreetLeverage for creating a forum where issues affecting sign language interpreters and the field of sign language interpreting can be raised and discussed thoughtfully and respectfully. This forum has allowed me the opportunity to share my communications with the RID Board on questions I believe need to be answered regarding the “enhanced” NIC certification test. My last posting on this topic resulted in a rich and stimulating discussion.

The questions raised in my post, Defenders of Certification: Sign Language Interpreters Question “Enhanced” RID NIC Test, and that will be raised here, are not raised out of nostalgic ties; not raised out of a desire to cling to the past; not raised out of a desire to foment dissent. In truth, I raise these questions to provide certified sign language interpreters (and, indeed, all members of RID) with the information and facts necessary for us to serve as defenders, not critics, of the current NIC evaluation. Sadly I believe that, to date, we have not been provided with such information or facts.

What follows is a letter I sent to the RID Board on June 25, 2012 in response to the “comprehensive report” that was released by the Board on June 7, 2012, which was in response to my March 18, 2012 letter questioning aspects of the “enhanced” NIC test.

My Intent

I want to explicitly state my intent in writing these letters and raising these questions is not to spur divisiveness but rather to garner transparency and to seek the underlying rationale for decisions that were made without full involvement of the certified membership. To date I have not received a response from the Board to this letter, although I have received an acknowledgement that my email letter has been received.

I truly hope that a meaningful response to the issues I have raised will be provided in a timely and public manner. If such a response is forthcoming, I would suggest that the approach of “It is because we say it is” is decidedly not a useful strategy in addressing an issue as important to RID certified members as this is; the response must be one that comports well with the more than forty years of experience RID has assessing interpreting competency and that comports with the real world experiences of practitioners.

My hope is that my postings on this issue can, and will, spur discussions that can better inform our decision-making process and, ultimately, improve our assessment of who is qualified to become “one of us”.

My Letter

June 25, 2012

To Presidents Prudhom and Scoggins and members of the RID Board:

Thank you for your letter dated June 7 and for the “comprehensive report” that was attached. Following you will find my response to the report and some additional issues/questions I have that were raised by the report.

You are correct that my letter was made public. But, please know that it was made public only after more than a month had passed after I had sent the letter to all members of the Board during which time its receipt was not even acknowledged by the Board. While I appreciate the demands on a volunteer Board, I would suggest that the serious questions I raised deserved a much more timely response (and at least the courtesy of an acknowledgment of receipt by the Board) and the answers which I requested should not have taken three months to assemble. I also believe that the “community discussion” its release created is definitely a positive discussion and one that should have occurred as the “enhancements” were conceived and most definitely before they were implemented. I hope that continued, open and public discussion will only help to strengthen our decisions and more fully engage the RID membership in these critical decisions and programs. I must tell you that I do intend to post this letter to you in a public forum in a couple of weeks – I firmly believe that providing a forum for members discuss these issues not only allows a fuller airing of issues, but also allows the Board to have a more accurate picture of the pulse of the membership than it appears to have on this issue.

I am sure you can appreciate the fact that the report that was generated in response to my letter is, based on the numerous emails I have received and numerous postings on social media, viewed by many as quite a defensive, reactive and inadequate response. My intent here, and throughout, is not to create an atmosphere of confrontation or to incite divisiveness. Rather, my intent is that, unlike decisions taken in the past, we can all agree that the decisions we make in the certification program are based on empirical data, not the feelings or beliefs of a small group that may not well-represent the membership. In your letter to me you state “NAD and RID fully support the work and direction of the enhanced NIC…”. I humbly and respectfully strongly disagree. At the very least in the case of RID, I think it is much more accurate and truthful to state that the RID Board fully supports the “enhanced” NIC. I say this because there is considerable discontent and unrest among the membership regarding the “enhanced” NIC. I also believe there has been anything but acceptable and appropriate levels of transparency regarding decisions surrounding these “enhancements” which is quite surprising for what the Board claims is a “membership driven” organization. I think that it is fair to say that for many members there is a clear sense that critical decisions that define who we are as interpreters have been made without significant and meaningful involvement of, and engagement with, the members and most definitely without meaningful discussion at our regional and national conferences, again belying the notion of RID as a “membership driven” organization. I certainly urge you to make good on your pledge to “…dedicated communication effort…now and in the future” and hope you do so proactively rather than reactively.

The “comprehensive report” provided many statements that asserted that the “enhanced” NIC was valid and reliable but provided no empirical or psychometric data to support such assertions. Unfortunately I see no proactive involvement of the membership in many of the issues I have raised; I see no checks and balances built into the current “enhanced” NIC that can ensure impartiality and objectivity; I see no independent checks and balances of the design and implement work off the consulting group; I see no concern for the negative reactions and responses of members that this “enhanced” NIC is not based on sound empirical data; and unfortunately I see no evidence in the Board’s actions that can support its claim that RID, in this arena at least, is a “membership-driven” organization.

I fully appreciate the pressures and demands on an all-volunteer Board having been there myself. I trust you will accept my response and the questions I raise below in the spirit of moving toward a stronger and more cohesive RID.  However, as I stated before, the fact is that there is growing discontent among the membership with the manner in which decisions are taken within the organization and a growing feeling of complete disenfranchisement within the organization. There already is a growing number of members and an increasing number of states in the US who believe that RID certification has become meaningless and irrelevant. More importantly, there is an increasing chorus of RID members crying for an alternate organization and that chorus grows as members take the “enhanced” NIC. I, for one, am saddened to see that number grow. But if sound, logical answers to the questions I have previously raised and also raise below cannot be provided then, unfortunately, I would have no option but to join their number and advocate loudly and passionately for creation of an alternate organization. I would be incredibly saddened should that come to pass.

I thank you for your willingness to engage in this discussion and I am willing to continue this very important discussion in any forum you feel will be beneficial (email, VideoPhone, phone, face-to-face meetings, etc.).

I eagerly await your response.

Sincerely,

dennis

Dennis Cokely, Professor
Director, American Sign Language Program
Director, World Languages Center
Chair, Department of Languages, Literatures and Cultures
Northeastern University
Boston, MA 02115
 

My Response to the June 7, 2012 “Comprehensive Report” on the NIC “enhancements

1) Before raising specific questions generated by the report, I ask that the Board offer an explanation to the membership for why the same consultant firm that was directly responsible for the design and implementation of the “enhanced” NIC was also asked to write the interim report which asserts that the “enhanced” NIC is valid and reliable. While I, in no way, intend to cast aspersions on the Caviart group, I believe it is incumbent upon the Board explain why this is not a clear conflict of interest and why an independent psychometrician was not engaged to review the overall process and write the report? Given the catastrophic recent NIC history and the absolute appearance of a possible conflict of interest, I would urge the Board to address this as quickly as possible. Why should we have faith in the validity of the Caviart report? Why is this not a conflict of interest given that Caviart has a vested interest in convincing the Board and members that its initial test development is valid and reliable?

2) It comes as quite a surprise to me and I am sure to most RID members that, on page #2 of the report, we learn that the profile of a “certified NIC (level 1) interpreter” was developed by 14 members of the NIC Task Force, an NIC Scoring Group (whose composition is unknown) and an unidentified group of “subject-matter experts”. This means that, if one includes the Board (which, I assume, approved the profile), it would appear that fewer than 100 members had a hand in determining this profile. This represents less than .009% of the membership. Why was this profile not circulated to the membership for comment and input? How is not circulating this profile consistent with claims of a “membership driven” organization?

3) It is also troubling is that the profile of a “certified NIC (level 1) interpreter” is offered without any rational, explanation or justification. What empirical basis is there to support this profile? What data is there to suggest that a “…vocabulary at a first year undergraduate level…” is appropriate? What is meant by “…quickly adapt to different situations and communication styles”?

4) But what is even more troubling on page #2 is the incredibly dismissive tone in the report intended for those who might question or challenge (“That is fine.”). While it is certainly true that “…no profile will satisfy everyone”, I believe that the leadership owes it to the members to create an environment in which questions and challenges are welcomed because, presumably, the leadership has data to support its choices and decisions and thus can appropriately respond to such questions and challenges. The tone here smacks of “it is because we say it is”. If this tone emanates from the consultant group then perhaps we have retained the wrong group. If the tone emanates from the leadership, then the situation is even worse that I thought and we definitely have a crisis of leadership. How is such a tone consistent with a “membership driven” organization? Does the membership not, at least, deserve the courtesy of opportunities to discuss the definition of the threshold that marks who is “one of us”?

5) Also, on page #2, it appears that a decision has been made that there will be “levels” to the NIC (“currently called Level 1”). If this is indeed the case, this represents a significant departure from past practice (the recently aborted iteration of the NIC notwithstanding) in which we have had a “generalist then specialist” approach to certification. I ask that the Board release the overall comprehensive master plan mentioned on page #3 for the certification program as well as the specific criteria for determining “…higher and specialized levels”. What is the overall master plan for the certification program?

6) Again on page #2, I believe it is totally inaccurate to make the claim that “…this statement summarizes what both organizations believe…”. At the present time I believe that it is accurate to say, in the case of RID at least, that this statement summarizes what the Board, the NIC Task Force and an NIC Scoring Group whose composition is unknown believes. The membership were not at all engaged in this discussion nor invited to provide feedback on this issue.  Can the RID Board explain how it purports to represent the membership on this issue when it has not fully engaged the membership?

7) On page #3, it is extremely troublesome to read that the claim “…more interpreter/consumer interactions occurred remotely…” is based on the feelings of the NIC Task Force rather than on empirical data. In fact, according to a recently completed survey of RID Certified and Associate members conducted by the National Interpreter Education Center, 70% of interpreters who responded report they do absolutely no work in VRS and 92% do no work in VRI. These data make it clear that it is still the case that from interpreters’ experience, the overwhelming majority of the work that interpreters do is conducted in face-to-face interactions.  While it may be that Deaf people are using more VRS/VRI, at this point in our history the fact is that the vast majority of interpreters do not work in those settings. This begs the question of why we developed an assessment approach that appears predicated on beliefs, settings, assumptions in which only 30% and 8% of us work?

8 ) Given the fact that some members of the NIC Task force have VRS/VRI ties, one also has to wonder again about the wisdom of relying on feelings rather than empirical data. Why are feelings used over empirical data? Again I do not intend to cast aspersions, but when segments on the “enhanced” NIC have durations of three to five minutes (which resemble durations found in VRS calls), one wonders what response the Board might have to the fact that there is the appearance of a conflict of interest for members of the NIC Task force who do have VRS/VRI ties?

9) On page 5 the process by which the “content and format” of the “enhanced” NIC is described. The logic of this process seems, to me, to be somewhat circular and rather flawed. If I understand the process correctly, raters used the previous NIC exam, material with which they were intimately familiar having viewed the material dozens and dozens of times. They were then asked to identify portions of that material that they believed “…most effectively discriminated the skills of interpreters at the level described”. From a linguistic and sociolinguistic perspective, what is critical is that they selected these portions from within a larger context and those portions were taken from within a larger sample of a candidate’s work. Thus raters had significant background information influencing the choice of segments that they believed “…most effectively discriminated the skills of interpreters at the level described”.  Scoring criteria were then identified using a “proven algorithm”. While the algorithm may be “proven”, the elements (and the empirical support for those elements) to which the algorithm was applied need to be known. This is especially true since the previous iteration of the NIC listed rating elements that conflated language elements (e.g. articulation, use of space) with interpretation elements. What are the elements to which raters will apply the scoring algorithm and what is the empirical basis for those elements?

10) Having demonstrated that portions from within a larger context and taken from within a larger sample of a candidate’s work could be reliably rated, how then is it reasonable to conclude that portions generated with no larger context and from no larger work sample could or would suffice as a valid indicant of a candidate’s competence?

11) Having the Scoring Group finalize the profile, develop the scoring criteria, rate previously rated samples and then discuss their holistic ratings, places enormous and unchecked power in the hands of a small group. Why was there no independent confirmation of the Scoring Group’s work? Why were not different groups engaged in segments of this undertaking? How large was the Scoring Group? Who were its members?

12) On page 6 we are told that the final vignettes were “…believed, by the Scoring Group, to have the most appropriate content, vocabulary and stimulus material”. The Scoring Group then developed new scoring criteria – again enormous and unchecked power in the hands of a small group. Is there a rationale for this? Are we to believe that there is a different set of criteria for each vignette? If so, what is the full set of criteria used to rate a candidate and what is the empirical basis for each criterion? Why should we believe the Scoring Group is qualified to make such determinations?

13) I submit that the assertion that “While content validity is critical, face validity is not” critical completely ignores the recent past with regard to the NIC and reflects a complete lack of understanding of our assessment and institutional history. Certainly face validity is unquestionably important for market acceptance. And most certainly one incredibly important segment of that market is the RID Certified and Associate membership and students in Interpreter Education Programs. If certification is to have any value, these stakeholders simply must feel and believe that the high-stakes assessment that is the NIC, at least looks like what interpreters do regularly. While it may be true that an assessment of interpretation skills does not have to look like what interpreters do regularly, one would think that, given the last significantly flawed iteration of the NIC, the Board would most definitely want the assessment to look like what interpreters do regularly. Given the incredibly negative issues surrounding the last iteration of the NIC, why would the Board, the NIC Task Force and the NIC Scoring Group endorse an assessment approach that most clearly lacks face validity and seems not to be widely supported?

14) On page 8 the dismissive tone of this report continues with the assertion that there is “…absolutely no merit to this suggestion.” In the present climate, I assert precisely the opposite – if the members of RID do not believe that the assessment is valid and looks valid, if we cannot defend it, if we cannot/will not encourage those who are not yet certified to seek certification, then the assessment process is seriouslyflawed. I believe that there is incredible merit to the need for face validity of the NIC. The report asserts that the “enhanced” NIC “…has significant face validity…”, but this is just another example of “it is because we say it is”. No empirical data is offered here. However, candidates who have taken the “enhanced” NIC and who have contacted me almost unanimously say that they think that the “enhanced” NIC did not fairly sample their interpreting skills, did not look like what they do on a regular basis and did not allow them to demonstrate what they do when they interpret. Can the Board, the NIC Task Force or the NIC Scoring Group provide empirical data that candidates do indeed feel the “enhanced” NIC fairly samples and assesses their work? What percent of those who have taken the “enhanced” NIC report that it “fairly sampled” their interpreting competence? I doubt such data even exists or is collected.

15) On page 8 the report states that panels were asked to “…identify the amount of time that it takes to accurately assess a candidate’s skill.” Again I ask whether there is any empirical data to support this approach; what we have is self-report data, drawn from an individual’s various experiences that are based on samples that are highly contextualized. If I state that in a given real-world context I can determine in two or three minutes whether I can accurately assess skill for this particular situation, I cannot validly apply that determination to other contexts. Why have we have taken timing information based on contextualized self-reports and applied that timing information to de-contextualized vignettes? Is there any empirical support for doing so? Are we asking raters trained to assess the former NIC to make this determination? If so, by what justification? Or is this another instance in which the ubiquitous Scoring Group

16) The unidentified subject matter “experts” (number and qualifications unknown) believe that “niceties” can be excluded because they “…provide little information about a candidate’s skill”. It certainly is true that, from an interpreter’s perspective, the beginnings of interactions are often not challenging and thus may not be fully representative of a candidate’s skill set. However, there is clear linguistic empirical data to show that these “niceties” are often essential to an interpreter’s overall ease, comfort and comprehension of a speaker/signer and thus important to rendering successful interpretations. Indeed, one cannot “…go right to the heart of the communication encounter.” Linguists and Sociolinguists have shown clearly that successful communication is an evolved and negotiated interaction; one cannot properly and fully understand “the heart” of an interaction while ignoring or not having access to the “skeleton” and the “flesh” that surround the “heart”. We return again to the face validity question. How is it that we can warrant that those who pass the NIC can “…relay the essence of a message…” when our assessment strips away all of the context, linguistic background and interactional unfolding that leads up to “…the essence of a message…”? What is the empirical data to support this decision? Certainly the “comprehensive report” does not address this question.

17) The use of “tower crane operators” is an insulting and ignorant analogy at best (no offense intended to tower crane operators). Granted there is considerable pressure in being a crane operator. However, there is also a clear “right and wrong” result, the ball is manipulated correctly or it is not; the result is black or white. The results on a crane operator test are plain for all to see – the wall comes down or it does not (true some might be more efficient than others; but ultimately if the wall does not come down the operator has not been successful). However, as any interpreter knows, interpreting is anything but a “black or white” cognitive task. There are a finite number of moves that are possible with a tower crane. However, as any interpreter knows there are myriad possibilities for rendering successful interpretations in any interaction because we are dealing with people not machines. And while the obstacles through which crane operators must move to demonstrate their skill are finite, interpreters clearly know that because communication involves a range of human beings, a range of situations, and a range of communicative goals, the obstacles through which we must maneuver are virtually infinite. It does not follow logically that because tower crane operators can be assessed by an examination that lacks face validity that interpreters should accept, much less endorse, any examination that lacks face validity. Why should we accept a lack of face validity?

18) The use of “police officers” is also not an appropriate analogy for our purposes. Police officers spend months and months training at the police academy; they do not get to take their performance exam until/unless they have performed well in their training routines during which they have fired their guns countless times. The police officer test itself is valid because it presumes months and months of training that is specific to being a police officer. However, in our case, the claim that NIC “…standards include requirements for education…”(pg. 6), make using police officer testing a false analogy. Our forthcoming “requirement for education” is, at this point, only a BA/BS degree but not a degree in interpreting. So, even though the police officer test may not be indicative of what they do on a daily basis, police cadets have had months of training and practice to demonstrate what they will do on a daily basis. RID has no such warrantied, specific interpreter educational background requirement for NIC candidates. Lacking the focused, specific training/education that is required of police officers (and tower crane operators) and that forms the foundation on which their tests can rest, why would we adopt an assessment approach that presumes such focused, specified training/education when we do not yet have such focused, specified training/education?

19) On page 9 the report asserts that a testing program is fair if “…every candidate has the same opportunity to demonstrate his or her competence.” Again, based on the numerous comments I have received from those who have taken the “enhanced” NIC, the majority feel they did not have an opportunity to demonstrate their competence. All candidates may be presented with the same logistically structured opportunity with the NIC, but if that opportunity is believed by a majority of candidates not to afford them an “…opportunity to demonstrate his or her competence” or is believed by them to be a flawed opportunity, how can such a program be judged as fair? How does the Board respond to candidates who feel they did not have an opportunity to demonstrate their competence?

20) Page 9 states that there is “…no evidence…that suggests that physical endurance is required for the job.” This patently ignores significant research in signed and spoken language interpreting research on this matter.  It patently fails to realize that physical endurance is not the critical issue in interpretation — cognitive endurance definitely is!!!! This is extremely well documented in the literature. With the “enhanced” NIC vignettes ranging from only three minutes to five minutes in length (although one LTA emailed me saying that they carefully timed each scenario and the actual range was between 1:20 and 3:20!!), it is virtually impossible to see how we assess cognitive endurance. This is, after all, one of the most important reasons why we advocate for the use of teams in many situations. If there is no evidence in the “Role Delineation Study” that speaks to cognitive fatigue, then I suggest that that study is seriously flawed. If there is such evidence in the Study and the “enhanced NIC” ignores this, then the program is seriously flawed. Why do we think that cognitive fatigue is not a critical factor to assess? And why was the “Role Delineation Study” not more widely vetted and shared?

21) Page 10 states that “…RID has carefully specified the testing conditions…”. Based on information I have received from candidates who have taken the “enhanced” RID, this means that candidates must be seated and must remain so for the duration of the test. As any interpreter knows, when an interpreter is seated, his/her range of motion is severely restricted and thus his/her use of space for semantic/linguistic purposes is also restricted. Given that we have never restricted candidates in this manner, what empirical evidence is there that placing interpreters in such restricted conditions will produce samples of their work that are indicative of their overall competence? Why would we want to restrict/constrain the use of semantic/linguistic space?

22) Page 11 proclaims the hope that the “enhanced” NIC will “…earn the value and respect from consumers that it deserves”. I submit that the “enhanced” NIC cannot earn respect from consumers until and unless it is accepted, embraced and valued by practitioners. This status report references a number of reports and studies that, to my knowledge, have not been made available to the RID membership. When can members expect release of all reports that are referenced in the Status Report?

My Previous Questions

In my initial letter to the Board, I asked nine questions. I was told that a “comprehensive report” would be issued that would address these questions. Unfortunately, I do not believe that any of the questions has been answered satisfactorily.

1) RID members need a more thorough explanation of why time and a simple mathematical formula should be the primary drivers behind the format of the certification test; if this is not true, then a clear explanation should be provided for how the current 4-minute per vignette test segmentation was determined.

Answer provided in the “comprehensive report” is inadequate

2) An explanation for the process/principles used in the selection of and/or development of the vignettes be made known to the membership.

Answer provided in the “comprehensive report” is inadequate

3) A full explanation of the empirical justification for this 4-minute approach must be provided to the membership.

Answer provided in the “comprehensive report” is inadequate

4) A clear explanation of the rationale and justification for placing candidates at such an interpreting disadvantage must be provided to the membership.

Answer provided in the “comprehensive report” is inadequate

5) A detailed explanation of the rational for, and empirical support for this decision and this deviation from forty years of experience is also needed by the membership.

Answer provided in the “comprehensive report” is inadequate

6) If there is evidence that supports the claim that a 4-miute sample can validly and reliably assess a candidate’s ability to assess sustained quality over time, then it must be made known to the membership.

Answer provided in the “comprehensive report” is inadequate

7) What are the various English and ASL grammatical and semantic features in vignettes that raters will be assessing and do these five 4-minute vignettes provide sufficient linguistic and discourse variation to elicit an appropriate range of English and ASL grammatical and semantic features?

Answer provided in the “comprehensive report” is inadequate

8 ) Since using discretion in selecting assignments is one of the core operating principles of our long-standing Code, the rationale for adopting an “all or nothing” approach must be made clear to the membership.

Answer provided in the “comprehensive report” is inadequate

9) A clear, empirically supported explanation of why the current NIC assessment is valid and can be reliably assessed by raters must be provided to the membership.

Answer provided in the “comprehensive report” is inadequate

Posted on 138 Comments

Defenders of Certification: Sign Language Interpreters Question “Enhanced” RID NIC Test

Sign Language Interpreters - Defenders of Certification

The advent of the tiered NIC test brought with it a host of questions and concerns among members of RID. Dennis Cokely shares a personal letter sent to the RID Board and outlines his requests for explanation regarding the format and procedures of the current certification process.

At this point in our history, the NIC assessment is the foundation for determining who is “one of us” and, as such, certified members of RID should be the defenders of the certification process. However, the fact that certified RID members are unsure of the validity of the current NIC assessment is unacceptable. I believe that the NIC Task Force and the Board of Directors have implemented changes to the RID assessment process the validity of which has not all been transparent to the certified membership. And so, instead of being defenders of the process, we find ourselves in the position of questioning, challenging and/or belittling the recent RID assessments procedures.

My Letter

On March 18,2012, I sent an email letter to each member of the RID Board of Directors in which I raised a number of questions regarding the new “enhancements” to the NIC test. That letter is reprinted below.

Before reading the letter, it is important to me that you understand the spirit in which that letter was sent.

My intent in sending the letter was neither to create or enflame divisiveness within RID nor was it to attack the current leadership of the RID. Rather it was a request that the Board provide the information necessary so that the RID membership, especially the certified membership, could feel confident and secure in the knowledge that the “enhanced NIC” was indeed valid and reliable; information that was not made available for the previous iteration of the NIC.

Until the day when RID (and we are RID) has a transparently valid and reliable certification process that determines who will be “one of us”, we will always have division and animus (parenthetically, I believe this can only be avoided if we, RID, decide to divest ourselves of the assessment process). My letter was sent to the Board requesting that all the information and documentation that provided the psychometric basis for the “enhanced NIC” be made available to all of the members. The Board has committed to releasing a report that would address the questions I raised.

RID Response

On April 22 I received an email from the RID President that stated, in part: “…the board of directors and national office staff agree the comprehensive report would be shared with the entire membership.  Therefore, this will take some time and resources to complete and request your patience and continued support to allow us the time to complete this comprehensive report. In fact, the work has been underway since the receipt of your letter.”

To be sure, it is unclear to me why the answers to the questions I raised should “…take some time and resources to complete.” After all the questions I raise are the essential questions one must ask and the evidence one must have in advance of implementing such a radically new assessment approach. The information should be readily available; if it has to be created in response to the questions I raise, there are even more serious questions about the process by which this iteration of the NIC was developed and implemented. Nevertheless, I applaud the fact that the RID Board will share full information regarding the new NIC with the membership. Hopefully that report will be issued in a timely manner and, in my opinion, it certainly must happen in advance of the regional conferences.

Reactions — Keep Them in Check

Given all of this, I trust you will read the following letter in the spirit in which it was intended. I sincerely hope that any reaction you may have will be held in check until we all receive the “comprehensive report” from the Board. I believe that any action prior to receipt of the “comprehensive report” would be premature and uniformed.

Letter Reprint

Members of the Board of Directors
Registry of Interpreters for the Deaf

333 Commerce Street
Alexandria, VA 22314 
 

March 18, 2012

To Members of the Board,

I am writing this letter to the Board, one of the very few I have written since 1972, as a concerned and dedicated member of RID for over forty years and as a Past President of RID. Specifically, I am extremely concerned about the new “enhancements” to the NIC test. I think it goes without saying that the last iteration of the NIC was significantly flawed. Claiming, as we did, (lacking both the sophistication and the empirical data) that a three-tiered certification based on a single evaluation test was valid and defensible, was clearly shown to be a serious mistake (one which we made earlier in our first effort at testing – CI/CT/CSC). With this latest unsubstantiated testing attempt, not only did we damage the credibility of the NIC and the RID itself in the minds of many RID members but perhaps more importantly in the minds of many Deaf people. Both interpreters and Deaf people saw that the test results and tiered certifications awarded often did not match the reality experienced by the “eyes on the street”.

I believe that the lesson that must be learned here is clear — we should definitely not advance an approach to testing that is not directly supported by empirical data on sign language interpretation and that we must make that empirical data clearly and widely known to interpreters and Deaf people.

Make no mistake, I applaud some of the changes to the NIC, specifically uploading a candidate’s video data to a secure server and having those video data available to be viewed by multiple raters. Unfortunately I believe we have made the same fatal mistake – lack of empirical data – with the newest iteration of the NIC as we made with the last iteration and as we made in 1972. Unless there is evidence that has not been made publically available, I believe that the current NIC testing approach lacks face validity — it does not look like what interpreters regularly do. Perhaps better stated, I believe the current test cannot claim to validly certify a candidate’s ability to interpret in a way that reflects real world practice. Certainly there is nothing in the research literature relevant to sign language interpreters of which I am aware that would support the current testing approach. I make the following statements and raise the following questions and concerns based on the new Candidate Handbook 2011 and on conversations with several candidates who have taken the current NIC.

1. It appears that someone predetermined that the test should last only an hour and then the resultant math determined that each of the two ethical and five performance scenarios would last only 4 minutes. If true, RID members need a more thorough explanation of why time and a simple mathematical formula should be the primary drivers behind the format of the certification test; if this is not true, then a clear explanation should be provided for how the current 4-minute per vignette test segmentation was determined.

2. I agree that that it may be possible to make a marginally valid, albeit shallow, determination of one’s approach to ethical decision-making and one’s knowledge of the Code of Professional Conduct from two 4-minute vignettes. However, one would hope that the vignettes are sufficiently complex that they will elicit higher levels of ethical thinking than mere regurgitation of the Code of Professional Conduct. A description of the guiding principles used to develop and/or select the ethical vignettes must be provided to the RID membership. Note I am not asking for the rating rubrics (I agree that teaching to the rubrics was a significant issue in the last iteration), I am simply asking that an explanation for the process/principles used in the selection of and/or development of the vignettes be made known to the membership.

3. I am aware of no research that provides evidence that a 4-minute sample of a piece of interpretation is sufficient to make a determination of overall interpretation competence. What the research does show is that during the first five minutes of a twenty minute monologue an interpreter’s work is often “less challenging” because it is the most predictable – introductions, niceties, setting an overall tone for a talk or meeting, etc. This is also true of the last five minutes of an interpreter’s work – summaries, next steps, closings, etc. Consequently, if all of the five performance vignettes were from the first five minutes of interactions, we would only be sampling and rating the “less challenging” parts of interactions and thus would not be presented with a true and valid representative sample of a candidate’s overall interpreting proficiency. I might agree that if we had five 20-minute samples of an interpreter’s work and we wished to select 4-minute samples from each 20-minute sample (some from the beginning, some from the middle and some from the end) then perhaps we might have a more thorough and more time efficient way of rating an interpreter’s work. But what we have here with the current NIC is clearly not 4-minute samples from longer samples of work. A full explanation of the empirical justification for this 4-minute sampling approach must be provided to the membership.

4. According to the Candidate Handbook, however, some of the vignettes will require that the candidate begin interpreting in the middle portions of interactions after providing the candidate with only a written synopsis of what has transpired up to that point in the interaction. Here again, I contend there is no empirical data that can justify this as a valid approach to obtaining a true and valid sample of a candidate’s overall interpreting competence. As any experienced interpreter knows, by the mid-point of any interpreted interaction the interpreter has developed some content background information (which I presume the NIC proposes to present in printed form). But more importantly the interpreter has a sense of communicative preferences, interactional rhythm, signing style, accents, spoken/signing speeds, prosodic features, etc. None of this can be presented in printed form in any manner that assists the candidate nor can it be presented in a manner that validly replicates what happens in real life.

On this basis alone, I would contend that this 4-minute assessment approach does not provide the essential cognitive, discourse or linguistic tools/knowledge that are available and that unfold in “real life” situations. Additionally, and perhaps more importantly, by the halfway point in any interaction the interpreter has acquired an “interactional schema”. As any experienced interpreter knows, this relates directly to critical areas such as over-arching goals, what counts as success and the overall interactional rhythm and flow. Absolutely none of this is accessible to a candidate suddenly instructed to begin in the middle of an interaction for which only written background content information has been provided. Of necessity, the written background will be about content, but none of this is what is most important to interpreters. A clear explanation of the rationale and justification for placing candidates at such an interpreting disadvantage must be provided to the membership.

5) Given that each performance vignette provides only 4 minutes of a candidate’s work, it would appear that we, as an organization, are no longer concerned about the ability to sustain quality of work during an interpreted interaction. For the past forty years the RID evaluations have contained interactions (monologues and/or dialogues) that have lasted 15-20 minutes in length. This was essentially due to the fact that this most closely reflected the real world work and experience of interpreters and then raters could sample within interactions, not across what are essentially 4-minute, flawed interactions. A detailed explanation of the rational for, and empirical support for this decision and this deviation from forty years experience is also needed by the membership.

6. Given that each performance vignette provides only 4 minutes of a candidate’s work, it would appear that we, as an organization, are no longer interested in the ability to produce work of sustained quality over time. Clearly, a 4-minute text simply does not allow time for the candidate to demonstrate or time for the rater to assess meaning sustained over time. The rater has no opportunity to assess features such as consistent use of grammatical features (manual and non-manual), consistent use of space, consistent use of deitic markers, etc. Simply put, a 4-minute sample simply does not provide sufficient opportunity to demonstrate a candidate’s ability to sustain quality work over time. If there is evidence that supports the claim that a 4-miute sample can validly and reliably assess a candidate’s ability to assess sustained quality over time, then it must be made known to the membership.

7. With a 4-miute segment to assess, the question must be asked “What are the raters looking for?”. It is clear that there is a new rating paradigm (pass/marginal pass, fail/marginal fail) and one could make a solid case for this. Certainly raters for the signed portions should be looking for grammatical features such as agreement, consistent use of “nonce signs” (signs established for this situation only), the use of coordinated and reflexive space, etc. But it is unclear what raters would be asked to assess in a 4-minute sample of work. Certainly raters are unable to assess the full range of linguistic competencies that interpreters must posses in order to able to interpret (if there evidence to support this it must be made public).  What are the various English and ASL grammatical and semantic features in vignettes that raters will be assessing and do these five 4-minute vignettes provide sufficient linguistics and discourse variation to elicit an appropriate range of English and ASL grammatical and semantic features?

8. As was true with the last iteration of the NIC we offer the candidate no opportunity to demonstrate the exercise of discretion. This clearly begs the question of whether there is any research that demonstrates that the five performance vignettes somehow represent “seminal” vignettes, i.e. vignettes for which no candidate would ever deem that he or she was an unsuitable fit. Clearly the message sent to candidates taking the NIC and to interpreters in general that one “must interpret everything presented to them” stands in stark contrast to our long held organizational belief that discretion in accepting assignments is critical. Since using discretion in selecting assignments is one of the core operating principles of our long-standing Code, the rationale for adopting an “all or nothing” approach must be made clear to the membership.

9. Virtually all of the candidate’s with whom I have spoken have the same reaction and response to the 4-minute performance vignettes. They state “They [the vignettes] were too short”; “I was just getting warmed up”; “I didn’t have the right information to start in the middle [of a vignette]”; “I don’t think it was a fair sample of my work”; “I needed more time to get over my nerves”; “This isn’t what I do everyday”. These comments are, to me as I hope they are to you, extremely troubling. Even if we assume there is a valid and reliable empirical basis for the “4-minute vignette” approach, the experience of the candidates is quite at odds with that basis. The danger here is that the candidates will, rightly or wrongly, begin to spread these perceptions to certified and not-yet certified interpreters. The end result will be that we return to the set of circumstances that resulted in abandoning the former iteration of the NIC – acting in the absence of empirical data to guide our decision-making. A clear, empirically supported explanation of why the current NIC assessment is valid and can be reliably assessed by raters must be provided to the membership.

The issue of how and the process by which we determine who will be viewed “as one of us” (i.e. who is certified) is of grave concern to many in the membership. As you should well know, it has clearly created some very, very deep rifts within the organization. So deep are the rifts that there is on-going discussion of creating an alternate organization. Yet, we in RID continue to move forward without the necessary empirical support we need to offer a credible approach to the testing process. The “alphabet soup” of certification that we have produced sadly moves us closer and closer to being quite laughable in the eyes of those who view professional organizations as knowing clearly how to determine who will be viewed as “one of us”.

In an ideal world, we would out-source the testing process so that RID could be the “assessment watch-dog” and thus RID could avoid any appearance of conflict of interest. Lacking that possibility at the present time, I believe that the Board should muster the political and moral will to insist on a truly valid and reliable certification test, accepted by the certified members. Then the Board should declare a phased in process by which ALL former certificates (save SC:L and CDI) would be declared invalid and no longer recognized. A staggered timeline would be put in place by which ALL those holding any certificate prior to the valid and reliable test would have to be retested and the “alphabet soup” would eventually no longer exist.

But we are where we are and that is that we have the current iteration of the NIC.

On behalf of the membership and all those who have served in positions of leadership, I am asking for a much greater level of transparency regarding the crafting of the current iteration of the NIC. If there is research data to support the decisions underlying the format of this iteration of the NIC those data must be made very public. I, for one, need to see the consultant’s report on why they believe this approach/format is valid and reliable before I can support this approach. I know that many of my colleagues, who are both members and organizational leaders, feel the same way.

Please know that I raise these questions and ask for this unprecedented level of public transparency in the best interests of RID the organization, of RID members and of Deaf people. I am happy to discuss any of these questions and concerns with the Board, individual or collectively, and/or the psychometric consultants hired to oversee the new NIC test.

Please let me know if you have any questions or need further clarification on any of the issues/questions raised. I eagerly await and expect your response to the questions and issues I have raised in this letter in a timely manner.

Sincerely
 
 
Dennis Cokely
Director, American Sign Language Program
Director, World Languages Center
Chair, Department of Languages, Literatures & Cultures

 

Overall Frame

We should definitely not advance an approach to testing that is not directly supported by empirical data on sign language interpretation and that we must make those empirical data clearly and widely known to interpreters and Deaf people

The Questions that Need Answers

1. RID members need a more thorough explanation of why time and a simple mathematical formula should be the primary drivers behind the format of the certification test; if this is not true, then a clear explanation should be provided for how the current 4-minute per vignette test segmentation was determined.

2. An explanation for the process/principles used in the selection of and/or development of the vignettes be made known to the membership.

3. A full explanation of the empirical justification for this 4-minute approach must be provided to the membership.

4. A clear explanation of the rationale and justification for placing candidates at such an interpreting disadvantage must be provided to the membership.

5. A detailed explanation of the rational for, and empirical support for this decision and this deviation from forty years experience is also needed by the membership.

6. If there is evidence that supports the claim that a 4-miute sample can validly and reliably assess a candidate’s ability to assess sustained quality over time, then it must be made known to the membership.

7. What are the various English and ASL grammatical and semantic features in vignettes that raters will be assessing and do these five 4-minute vignettes provide sufficient linguistic and discourse variation to elicit an appropriate range of English and ASL grammatical and semantic features?

8. Since using discretion in selecting assignments is one of the core operating principles of our long-standing Code, the rationale for adopting an “all or nothing” approach must be made clear to the membership.

9. A clear, empirically supported explanation of why the current NIC assessment is valid and can be reliably assessed by raters must be provided to the membership.