How the ODP Compiler Works, Part 1
Jun 30, 2019, 1:54 PM
- Next Project: ODP Compiler
- NSF ODP Tooling 1.0
- NSF ODP Tooling Example Project
- NSF ODP Tooling 1.2
- How the ODP Compiler Works, Part 1
- How the ODP Compiler Works, Part 2
- How the ODP Compiler Works, Part 3
- How the ODP Compiler Works, Part 4
- How the ODP Compiler Works, Part 5
- How the ODP Compiler Works, Part 6
- How the ODP Compiler Works, Part 7
A year ago, I started a project to compile NSFs - and particularly large XPages projects - independently of Designer. Since the initial releases, the project has grown, gaining an ODP exporter, extra Eclipse UI integration, and the ability to run without installing components on a remote server. It's become an integral part of my workflow with several projects, where I include the NSFs as part of a large Maven build alongside OSGi plugins and a final distribution.
Building this tooling required learning a lot about the internals of XPages, the specifics of how various design elements are stored and handled in an NSF, and miscellaneous bits about Equinox and Maven. Since there's a good amount of arcane knowledge embedded in the project, I think it'll be helpful to take some time to dive deep into what's going on, starting with XPages.
XSP to Java to Bytecode
The first challenge for me to overcome was how to go from XPages XML source to the Java class files (like those seen in the "Local" source folder in Designer) and finally to compiled Java bytecode. Much like in Designer's process, the middle part is just incidental: only the XSP source and the bytecode are actually stored in the NSF.
The official XSP -> Java compiler exists only in Designer, and so one route would be to try to get those plugins working on Domino. I think that'd work, but it'd be a huge hassle and, fortunately, an unnecessary one. The XPages Bazaar project contains essentially a clean re-implementation of the glue code required to coax the runtime into emitting what I needed. I used the Bazaar as an incubator for the early versions of the compiler, and tweaked the core code with additions and fixes to work with various needs I ran into.
On its own, the Bazaar did the job well of taking an XPage and compiling it with whatever the surrounding environment had. However, to compile a full XPages app as part of a Maven build, I'd need the ability to dynamically load XPages libraries and dependencies on the fly.
To do this, I added the option to include an update site directory, and then I have the compiler stream through all the plugins and initialize them. Fortunately, the OSGi environment makes this pretty easy. The
BundleContext object that's available from each OSGi bundle has an
installBundle method you can use by pointing at a bundle file URL, and then you can call
start() on that result to actually initialize it. I had to do a little extra work to account for bundles that shouldn't be started (source-only bundles and "fragments", which are additions onto normal plugins) and the like, but it's not too complicated.
Just installing the OSGi plugins isn't enough to get the XPages runtime to know about any libraries that may be included, however. This is done by finding all of the library extension contributors, sorting them for compatibility, wrapping them in a
com.ibm.xsp.library.LibraryWrapper for some reason, wrapping that in a
com.ibm.xsp.registry.config.SimpleRegistryProvider, and then adding the results of that to an in-memory
This is one of those cases where the actual code involved in the end isn't terribly long, but the amount of delving into the framework to figure out the needed parts was immense. In essence, each XPages application is represented as a
FacesProject implementation object, which retains a registry of the libraries it knows about. In a normal running server, this happens automatically during initialization: the runtime opens the NSF, figures out which libraries it needs, finds them from the ones it knows about, and constructs the app's contained world. For the compiler, I ended up having to do something of an ad-hoc version of this as it goes, finding all the parts that need to be kicked to notice the available libraries and have them around to map something like
<abc:someCustomComponent/> to an instance of
Custom controls are a similar story, but they use a specialized variant of the "real" library system called
LibraryFragment. The way these work, before the actual XSP -> Java compilation step, the compiler reads their definitions and adds them to the in-memory
FacesProject. It's important to define them all in this way before actually trying to interpret their source (or any XPages), because this allows for the interpreter to understand a reference to one CC from another. Fortunately, the process is separated enough that the definitions can all happen before any of the Java classes actually exist. Otherwise, I would have had to either try to make a dependency graph of which CC references which other, or just keep trying to compile them repeatedly until it got the right order by brute force. The latter process will actually show up in a future blog post.
Java Source Files
Compiling individual Java source files - both those that show up in the
Code/Java part of an NSF (or a custom source folder) as well as the translated XSP source - is handled by the Bazaar, in a class called
JavaSourceClassLoader. This class wraps around the in-memory Java compilation capabilities that were added to the JDK in Java 1.6. In particular, this class, paired with
SourceFileManager, provides knowledge to resolve classes from OSGi bundles in the active environment, including specialized knowledge of dependency management based on re-exported dependencies, embedded jars, and other OSGi-isms that play a big part in XPages libraries.
In essence, these classes provide a similar compilation environment to what Designer does when it creates the Eclipse project for compiling an NSF. They build up an environment with knowledge of all the dependencies that, in Designer, show up in "Plug-in Dependencies", and then the compiler feeds it all of the Java source files in the project. With all of this environment provided, the underlying Java compiler is able to do its job of converting them to bytecode en masse, which is then passed back to the compiler for insertion into the NSF.
Other Big Components
In the next blogs posts in this series, I'll go over some of the other big hurdles I had to overcome to get everything working properly: namely, figuring out all of the specialized behavior necessary to flag imported/created design notes properly and then the arcane incantations necessary to get this OSGi environment working outside of the Domino server.
Cameron Gregor - Jun 30, 2019, 9:38 PM
Hi Jesse, thank you for the work you are doing on this, I haven't had a real attemp at using yet but will be looking at in the next few months. It looks like it could be a good direction for us to go in.
In regards to Compiling XPages and custom controls from within eclipse. Do you think this could be done as an incremental eclipse builder? What I mean is, is the ODP Compilation currently done as a 'full build' of turning an ODP into an NSF? or can the xpages and custom control compilation be isolated as it's own builder?
Also in regards to XPages, do you think an XPages design could be delivered from within a plugin? I am imagining a scenario similar to the Single Copy Xpage design.
So Imagine I developed a 'SCXD template' of XPages and custom controls from within eclipse, compiled and delivered in a plugin instead of an NSF, could the XPages runtime be tricked into loading the xpages design from a plugin, in the same way it is tricked into loaded an SCXD from another NSF?
WK - Jul 1, 2019, 9:02 AM
>> Also in regards to XPages, do you think an XPages design could be delivered from within a plugin? I am imagining a scenario similar to the Single Copy Xpage design.
>> So Imagine I developed a 'SCXD template' of XPages and custom controls from within eclipse, compiled and delivered in a plugin instead of an NSF, could the XPages runtime be tricked into loading the xpages design from a plugin, in the same way it is tricked into loaded an SCXD from another NSF?
I decompiled some of xpages classes and unfortunatelly Modules Dynamic Class Loader has a lot of hardcoded assumptions about where the xpages live. I think at this point it is not possible out of box. You would have to swap class loader for something custom. I managed to do this for my dev environment - at this point I am able to load classes compiled by eclipse through swapped class loader (byte buddy is Your ... buddy here :)) which allows me to do incremental build of gigantic projects in seconds. But I would never do this for production because this would throw whole security out of window.
Jesse Gallagher - Jul 1, 2019, 9:59 AM
Incremental compilation is something of a wish-list item. The skeleton of the NSF ODP project could support a persistent runtime for as-you-go compilation, but currently the ODPCompiler class is geared towards a single full pass. I'll probably go into that a bit more in later parts of this series.
For running XPages from a plugin: as WK said, there are some baked-in assumptions. When I did my projects to run XPages outside of Domino, I created effectively two "modes" for running them: traditional web app and from-an-NSF. In the former, an XPage is essentially like a JSP or JSF page mapping a URL like /foo/bar/baz/SomePage.xsp to a class name of xsp.foo.bar.baz.SomePage. However, that lacks some critical capabilities of the from-an-NSF mode - things like sessionAsSigner, for example. Some of those can be shimmed around by providing contextual variables (I believe that the normal mechanisms to get the current session, database, etc. check if they're defined first before trying to derive anew), but some make direct assumptions about the context. So, essentially, "yes and no".
Cameron Gregor - Jul 1, 2019, 8:24 PM
Thanks WK and Jesse. I will put my dream on hold then. if only it was open source!
One other idea I had, is we have a bunch of global custom controls that are used in many applications.
I don't like the design inheritance method of updating these as it doesn't suit our style of deployment. So at the moment I am copying them around the templates whenever there are changes.
I had a project called bootlegger, which was a designer plugin, which exported the java + xspconfig files to a plugin, and delivered them as actual Xpages Controls from a library.
This way, I could update them once and then automatically all nsfs would effectively have the updated versions. However this all wasn't very streamlined and it ended up going the lazy route and just went back to copying them around.
I was thinking, If I can at least author these xsp files within eclipse ide with an xml editor, and have them compiled to a specific package within the plugin, I should be able to deliver these global custom controls as Xsp Library controls.
So that is where I was thinking about using the xsp to java compiler within eclipse, maybe I will have a look at trying to develop this after August if you think it is possible