You’ve definitely heard about Xtext, the famous text modeling framework, community award winner . We are all looking forward to the new project management wonder: the release of Helios, upcoming on June the 23rd, which will include Xtext 1.0.0. In this article, I want do describe some aspects of integration of Xtext-based languages into IDE.

Introduction

Creating your own domain-specific textual languages in Xtext is really easy, and the authors in Northern Germany are doing a good job to make it even easier. Once you have a language, you want to process it and this means usually to transform your model into another representation. The facility responsible for this transformation is called generator and consists of a bunch of transformation templates (e.G. XPand) and some code executing them. On some event, the model is read in and the transformations are applied to produce code. Currently, there are two ways of triggering the transformation: you can put the calling infrastructure in a Modeling Workflow Engine file, which is a kind of batch script or you can write a build-participant, which react on the changes in the model. In this article I want to describe my experiences with the builder approach.

Examining initial builder

The base idea behind the builder approach is the fact, that Xtext provides an project builder, which watches for all Xtext resource changes (Xtext-based DSL models) and notifies the so-called XtextBuilderParticipants. These are registered using Eclipse extension point org.eclipse.xtext.builder.participant and implement the IXtextBuilderParticipant interface:

...
public interface IXtextBuilderParticipant {
    void build(IBuildContext context, IProgressMonitor monitor) throws CoreException;
}

public interface IBuildContext {
    IProject getBuiltProject();
    List<IResourceDescription.Delta> getDeltas();
    ResourceSet getResourceSet();
    void needRebuild();
}

In order not to start completely from scratch, let us examine the builder provided in the Domain-Model-Example, delivered with Xtext. In the build method the example builder processes as follows:

• checks that the project is a Java project
• finds the generation folder
• creates an Output
• creates an Outlet and registers it in the Output
• registers a post-processor managing Java imports on the outlet
• creates the Xpand execution context
• registers the Java Beans version of the metamodel
• processes the changes and analyses if the files have to be deleted or not prior re-generation
• for all changed model elements re-generates

Quite a lot of tasks, so there is a good reason, why MWE scripts are used for generation.

Using Project natures

The first improvement I created for the builder is the ability to distinguish DSLs. The problem of triggering all XtextBuildParticipants on the change of any Xtext-based resource is that the entire Workspace gets re-generated. In my scenario, we used three DSLs and a random change caused three builders to run, even if the change has been made in another project of workspace. In order to solve this problem I used the standard Eclipse way to distinguish projects: Project Natures.

The nature I created is used as a marker for the builder and is emtpy:

public class UiNature implements IProjectNature {
	public static final String NATURE_ID = "de.techjava.dsl.uiNature";
	private IProject project;

	public void configure() throws CoreException {
        }

	public void deconfigure() throws CoreException {
	}
	public IProject getProject() {
		return project;
	}
	public void setProject(IProject project) {
		this.project = project;
	}
}

Don’t forget to register it in the plugin.xml:

   <extension point="org.eclipse.core.resources.natures"
         id="de.techjava.dsl.uiNature"
         name="User Interface DSL Project Nature">
      <runtime>
         <run class="de.techjava.dsl.userinterface.builder.UiNature" />
      </runtime>
   </extension>

As the first line in my builder I check the presence of the nature in the current project and stop building if the nature is not present:

if (!context.getBuiltProject().hasNature(UiNature.NATURE_ID)) {
	// skip projects without UI DSL nature
	return;
}

Now, what is missing is the ability to add and remove the nature to/from the project. I liked the way how Xpand/Xtend Natures are configured (using toggle action in configure menu of the project pop-up menu). What you need is a ToggleAction that can be switched on, if the nature is not present and switched off, if the nature is installed.

public class ToggleNatureAction implements IObjectActionDelegate {
	private ISelection selection;
	@SuppressWarnings("unchecked")
	public void run(IAction action) {
		if (selection instanceof IStructuredSelection) {
			for (Iterator it = ((IStructuredSelection) selection).iterator(); it.hasNext();) {
				Object element = it.next();
				IProject project = null;
				if (element instanceof IProject) {
					project = (IProject) element;
				} else if (element instanceof IAdaptable) {
					project = (IProject) ((IAdaptable) element).getAdapter(IProject.class);
				}
				if (project != null) {
					toggleNature(project);
				}
			}
		}
	}

	public void selectionChanged(IAction action, ISelection selection) {
		this.selection = selection;
	}

	public void setActivePart(IAction action, IWorkbenchPart targetPart) {
	}

	/**
	 * Toggles sample nature on a project
	 * @param project to have sample nature added or removed
	 */
	private void toggleNature(IProject project) {
	// implemetation of toggle nature ...
	}
}

In order to make an action to a toggle-action the Eclipse Command Framework, Object Contributions and the Command Core Expressions are needed. The idea is to register two actions as object contributions on the IProject using the org.eclipse.ui.popupMenus extension point and define visibility of those to be complement to each other (if one is visible the other is not) based on the object state, which is the presence of a project nature. The actions used point to the same implementation class but have different labels. The menu path to pop-up menu Project > Configure is org.eclipse.ui.projectConfigure/additions. The visibility is defined based on the objectState:

  <extension point="org.eclipse.ui.popupMenus">
      <objectContribution adaptable="true"
            id="de.techjava.dsl.userinterface.ui.addNature"
            objectClass="org.eclipse.core.resources.IProject">
         <action
               class="de.techjava.dsl.userinterface.ui.action.ToggleNatureAction"
               id="de.techjava.dsl.userinterface.ui.AddNatureAction"
               label="Add User Interface DSL Nature"
               menubarPath="org.eclipse.ui.projectConfigure/additions">
         </action>
         <visibility>
            <not><objectState name="nature" value="de.techjava.dsl.uiNature" /></not>
         </visibility>
      </objectContribution>
      <objectContribution adaptable="true"
            id="de.techjava.dsl.userinterface.ui.removeNature"
            objectClass="org.eclipse.core.resources.IProject">
         <action
               class="de.techjava.dsl.userinterface.ui.action.ToggleNatureAction"
               id="de.techjava.dsl.userinterface.ui.AddNatureAction"
               label="Remove User Interface DSL Nature"
               menubarPath="org.eclipse.ui.projectConfigure/additions">
         </action>
         <visibility>
            <objectState name="nature" value="de.techjava.dsl.uiNature" />
         </visibility>
      </objectContribution>
   </extension>

Please note, that the nature is implemented and registered inside of the generator project and the toggle action is implemented in the UI project. Finally, the code of the toggleNature method, which is the standard code for the project nature installation and removal:

...
	IProjectDescription description = project.getDescription();
	String[] natures = description.getNatureIds();

	// remove the nature
	for (int i = 0; i < natures.length; ++i) {
		// if nature exists
		if (NATURE_ID.equals(natures[i])) {
			// Remove the nature
			String[] newNatures = new String[natures.length - 1];
			System.arraycopy(natures, 0, newNatures, 0, i);
			System.arraycopy(natures, i + 1, newNatures, i, natures.length - i - 1);
			description.setNatureIds(newNatures);
			project.setDescription(description, null);
			return;
		}
	}

	// Add the nature
	String[] newNatures = new String[natures.length + 1];
	System.arraycopy(natures, 0, newNatures, 0, natures.length);
	newNatures[natures.length] = NATURE_ID;
	description.setNatureIds(newNatures);
	project.setDescription(description, null);
...

That’s it, run and right-click on the Project and then go to the Configure Menu (pretty low in the pop-up) and you will see the result:

Parameterizing generator

Domain-specific languages should cover only the real requirements and incorporate the specific decision made in the project. So far the theory, but in fact it is a trade-off and a compromise, what is a part of the language and what is too specific to be covered. The same holds for the generator: on the one hand you want the generator to solve exactly your problem, on the other hand you want to make it generic enough to cover a class of similar problems.

I faced the problem of using the generator for two teams in a big project, where the langage could be reused, but the generation infrastructure need to be slightly modified. For example, the information about the generation target folder, package prefix and the fact which artifact should be generated differred between the teams. In the same time the teams were able to agree on the same language. In order not to develop two generators, I decided to parameterize the generator and the templates.

The Xpand2/Xtend developers provided a way of parameterization of the templates using the so-called Global Variables, using the GLOBALVAR keyword. My favorite way of using it is to create a cached extension for reading the variable:

/*
 * Retrieves the boolean value of the global variable set from outside of the template
 */
cached Boolean generateService() :
    GLOBALVAR generateService == "true"
;

In order to set global variables from the build participant the XPand execution context constructor takes a Map<String, Variable> argument. The nice story about the global variables, that you can provide any objects as values. So the advantage over the usage of MWE workflow with property files read-in during processing is the big flexibility in providing the object-graphs or even statefull objects as global variable values. A good example of usage of global variable is provided in Domain-Example with Java Import Tool:

JavaImportsTool importsTool = new JavaImportsTool();
...
ctx = new XpandExecutionContextImpl(output, null,
    Collections.singletonMap(JavaImportsTool.VAR_NAME,
    new Variable(JavaImportsTool.VAR_NAME,importsTool)),
    null, null);
...

In the same time it is important to be able to load simple properties from file in the generation project. For example you could implement your own ModelProperties container, which reads properties from the file located in the generation project. For example:

public class ModelProperties extends Properties {
	/**
	 * Indicates that no timestamp is available
	 */
	private static final long NO_TIMESTAMP = -1;

	private long lastModified = NO_TIMESTAMP;
	private final IFile propertyFile;
	private final HashMap<String, Variable> globalVars;

	/**
	 * Creates a valid model properties instance
	 *
	 * @param aProject reference to the current project
	 * @param name of the property file
	 * @throws CoreException on errors
	 */
	public ModelProperties(final IProject aProject, String filename) throws CoreException {
		if (aProject == null) {
			ErrorHelper.throwCoreException("project must be not null");
		}
		if (filename == null) {
			ErrorHelper.throwCoreException("filename must be not null");
		}
		this.globalVars = new HashMap<String, Variable>();
		this.propertyFile = (IFile) aProject.findMember(filename);
	}

	public IFile getPropertyFile() {
		return propertyFile;
	}

	/**
	 * Returns the global variables.
	 * @return global variables.
	 */
	public Map<String, Variable> getGlobalVars() {
		updateProperties();
		return globalVars;
	}

	@Override
	public String getProperty(String key) {
		updateProperties();
		return super.getProperty(key);
	}

	/**
	 * Updates the properties and the global variables if the property file has
	 * been modified.
	 */
	public void updateProperties() {
		if (this.propertyFile != null && lastModified != this.propertyFile.getModificationStamp()) {
			loadProperties();
			globalVars.clear();
			for (Map.Entry<Object, Object> entry : entrySet()) {
				final String propertyName = (String) entry.getKey();
				globalVars.put(propertyName, new Variable(propertyName, entry.getValue()));
			}
		}
	}

	/**
	 * Loads the properties from the model file
	 */
	private void loadProperties() {

		try {
			if (propertyFile != null && propertyFile.exists()) {
				super.load(propertyFile.getContents());
				this.lastModified = propertyFile.getModificationStamp();
				handleSpecialProperties();
			}
		} catch (CoreException e) {
			// no model property file isn't a problem but clear the properties:
			super.clear();
			this.lastModified = NO_TIMESTAMP;
		} catch (IOException e) {
			super.clear();
			this.lastModified = NO_TIMESTAMP;
		}
	}

	protected void handleSpecialProperties() {

	}
}

Using this class you can simple initialize the XPandExecutionContext with properties read from the file system. So in build-Method of your Generator put something like:

         ModelProperties structureProperties = new ModelProperties(context.getBuiltProject(), "structure.properties");
         XpandExecutionContextImpl ctx = new XpandExecutionContextImpl(output, null, structureProperties null, null);

Now you can resolve the values from the property files directly from the Templates and Extensions using the built-in GLOBAL VAR feature.