An extended version of Thushari Atapattu’s conference paper from CSEDU has recently been published as a chapter in a special edition of the book series Communications in Computer and Information Science.
The initial version of this paper was published at CSEDU in 2014, and received a nomination for Best Paper Award!
T. Atapattu, K. Falkner and N. Falkner. An Evaluation Methodology for Concept Maps Mined from Lecture Notes: An Educational Perspective. In Computer Supported Education, volume 510 of the series Communications and Information Science, 6th International Conference, CSEDU 2014, Barcelona, Spain, April 1-3, 2014, Revised Selected Papers, pp 68-83, December 2015.
Concept maps are effective tools that assist learners in organising and representing knowledge. Recent efforts in the area of concept mapping work toward semi- or fully automated approaches to extract concept maps from various text sources such as text books. The motivation for this research is twofold: novice learners require substantial assistance from experts in constructing their own maps, introducing additional hurdles, and alternatively, the workload required by academics in manually constructing expert maps is substantial and repetitive. A key limitation of an automated concept map generation is the lack of an evaluation framework to measure the quality of concept maps. The most common evaluation mechanism is measuring the overlap between machine-generated elements (e.g. concepts) with expert maps using relevancy measures such as precision and recall. However, in the educational context, the majority of knowledge presented is relevant to the learner, resulting in a large amount of information being retrieved for knowledge organisation. Therefore, this paper introduces a machine-based approach to evaluate the relative importance of knowledge by comparing with human judgment. We introduce three ranking models and conclude that the structural features are positively correlated with human experts (rs ~ 1) for courses with rich content and good structure (well-fitted).