Evaluating medical expert systems: what to test and how?

Many believe that medical expert systems have great potential to improve health care, but few of these systems have been rigorously evaluated, and even fewer are in routine use. We propose the evaluation of medical expert systems in two stages: laboratory and field testing. In the former, the perspectives of both prospective users and experts responsible for implementation are valuable. In the latter, the study must be designed to test, in an unbiased manner, whether the system is used in clinical practice, and if it is used, how it affects the structure, process and outcome of health care encounters. We conclude with proposals for encouraging the objective evaluation of these systems.