META EVALUATION PRACTICE IN AN ENGLISH DISTANT TRAINING PROGRAM IN JAKARTA

Abstract

Meta-evaluation is a means used to aggregate findings from a series of evaluations. It could be said like an audit of certain evaluation results using particular standards and criteria. The practice of meta-evaluation has not been widely carried out in the world, even more in learning programs organized by government institutions in Indonesia. This study is intended to evaluate the results of evaluation of a distant training program namely PJJ English for Customs Officers (EFCO Training) at Pusdiklat Keuangan Umum (General Finance Education and Training Center) or GFETC of Indonesia Ministry of Finance as the object of research. The study is expected to answer whether the evaluation of EFCO training meets the requirements of the utility, feasibility, propriety, and accuracy standards. Three meta-evaluators (1 internal and 2 external) gave a Yes or No tick to 290 items in the Meta Evaluation Checklist after they analyze the qualitative and quantitative evaluation data (documents) from the Reaction Evaluation (Level 1 Kirkpatrick Evaluation) and Learning Result Evaluation (Level 2 Kirkpatrick Evaluation) using Stufflebeam’s Program Evaluation Standards. The conclusion is that Level 1 and Level 2 (Kirkpatrick) evaluation of EFCO training met the standards of utility, feasibility, propriety, and accuracy. Another conclusion is that the implementation of meta-evaluation enables GFETC to indicate specific aspects of a training program that need to be improved in quality, and that meta evaluation can be used for quality assurance and decision making. The limitation of the research is that it has not been used to evaluate Level 3 and Level 4 of Kirkpatrick’s evaluation and that no panel review with stakeholders is conducted after the meta evaluation is completed. Recommendations of the research are the refined translation of standard items to check all the 4 levels of Kirkpatrick’s evaluation and that the research should be conducted for more training programs within longer period of evaluation.

Keywords: meta-evaluation, 4 level Kirkpatrick’s evaluation, utility, feasibility, propriety, accuracy