Step 4 Define the Metrics

In order to objectively measure performance, developers need to choose what specific elements are important to track. Key metrics need to be chosen and calibrated by a consensus of subject matter experts (in this case physicians) (24). These include a taxonomy/definition of errors and a method by which results can be integrated and reported for individual users as well as groups of users (25,26).

In 2001, Satava et al. organized a surgical skills workshop attended by an assembly of representatives from surgical societies and boards responsible for the education, training, and certification of surgeons. The primary purpose was to begin standardizing nomenclature and assessment methods so that the entire surgical education, training, and evaluation community could communicate more effectively and have a common basis to compare statistical results (27). The product of such a consensus could provide critical information to the developers of simulation technology. A similar consensus publication examining medical errors has been held and the manuscript is in progress.

Table 2 shows only an initial baseline of possible metrics that may make it possible to assess and assign the degree of competency for a trainee. However, the debate over proper metrics continues and undoubtedly the list presented in Table 2 will grow with time. For example, recent simulation discussion has focused on efficient use of OR resources (preoperative planning, equipment, ancillary staff if appropriate, gas/fluid management, amount of energy delivery), which may prove important for evaluating training over a variety of medical procedures.

With the collection and classification of such data, criterion levels for each key metric can be established to define the levels of performance listed in Table 3.

Expert systems that rely on knowledge or reasoning that emulates the performance of human subject matter experts can also be created. Such expert systems might be based on encoded rule statements that reflect individual or gathered expertise in a field.

The expert system would then display "reasoning" via a rule interpreter in order to reach a decision, come to a conclusion, or give up on a particular problem.

0 0

Post a comment