Eclipse committers' empirical comparison of language workbenches

Discussion of: Evaluation of Modeling Tools Adaptation, 2012, Amine El Kouhen, Cédric Dumoulin, Sébastien Gérard, Pierre Boulet [2]

In: Empirical Comparison of Language Workbenches, 2013, Steven Kelly, DSM'13 [1]

El Kouhen et al. [2] produced what may be the best empirical comparison of graphical language workbenches to date. They created BPMN support in each of five tools (RSA, GME, MetaEdit+, Obeo, and Eclipse GMF). Rather than students or makers of the tools, their users were the authors: Eclipse committers on the Papyrus project, which aimed to produce its own language workbench. They graded the tools on various features, but also on the total time taken to create the abstract syntax, concrete syntax and rules (Figure 1). The wide range of results is particularly interesting given the users’ familiarity with Eclipse GMF and ECore, which is also used in RSA and Obeo.



Figure 1: Days to implement BPMN [2]

A notable omission from the report is the lack of reference to related work. Had it been submitted for peer review this would no doubt have been corrected. The reason for not submitting these results as a publication, particularly after so much work, remains an open question.

The report is also a good example of the problem of guessing features which would contribute to effective usability of a language workbench. The authors spend three pages listing a variety of factors that they measure from the tools before the experiment to obtain a "usability" percentage, divided into "efficiency", "task visibility" and "visual coherence". The tool with the worst "efficiency" and "task visibility" percentages turned out to be the fastest, and the tool with the best "task visibility" percentage turned out to be the slowest.

Looking at the criteria, it seems they are more measures of how much the tools follow the user interface patterns familiar in Eclipse, rather than anything objectively good. Indeed the actual measured speed of use of the tools may point to a need to re-evaluate some user interface decisions that Eclipse users have become accustomed to expect. Similarly the common programmer’s desire for every detail of a task to be visible appears to run counter to productivity: e.g. only by hiding unnecessary details can third generation programming languages be more productive than assembly language – a principle surely familiar to all DSM practitioners.

References

[1] Kelly, S. 2013. Empirical Comparison of Language Workbenches.
SPLASH Workshop on Domain-Specific Modeling, Indianapolis, Indiana, 2013. http://www.dsmforum.org/events/DSM13/Papers/Kelly.pdf

[2] El Kouhen, A., Dumoulin, C., Gérard, S., and Boulet, P. 2012.
Evaluation of Modeling Tools Adaptation
. CNRS HAL. http://hal.archives-ouvertes.fr/docs/00/70/68/41/PDF/Evaluation_of_Modeling_Tools_Adaptation.pdf