Posts for thread "NSF ODP Tooling", by date ascending

Next Project: ODP Compiler

  • Mar 5, 2018

One of the larger thorns in my side with my Domino development lately has been trying to automate builds of on-disk projects into NSFs via Jenkins. In theory, the process is pretty straightforward. It even works sometimes! However, particularly once you add in the necessity to deploy OSGi plugins to Designer first and want to run it from Jenkins, things get extraordinarily flaky: Designer may not launch properly from a behind-the-scenes Jenkins runner, the plugin installation may mysteriously fail, and so forth - and the error reporting is difficult at best.

So it's been on my mind for a good while to find a way to get from an ODP to an NSF without involving Designer, and I decided over the last couple days to really take a swing at it. It's not a small task, though; the process involves a number of difficult steps:

  • Install and activate provided OSGi plugins
  • Create an XPages registry that knows about the XPages libraries installed on the server, including those just contributed
  • Translate XPages and Custom Controls into Java source, with intra-file knowledge of the just-added CCs
  • Create a Java classpath that matches the plug-in dependencies that Designer derives from the dependant XPages Libraries
  • Compile the resultant Java source and any Java classes in the NSF into bytecode
  • Recompose the composite data form of the file data for these elements and many file resources into their DXL ".metadata" files for import
  • Create an NSF and import all of this
  • Compile any LotusScript in source-based libraries from the ODP
  • Uninstall any hot-loaded OSGi plugins

Those steps even leave out some fiddly details, like components defined via .xsp-config files in the NSF or XSP-associated .properties files, not to mention any steps I haven't encountered yet. It's a lot of work!

My first hope was to be able to hook into the process that Designer uses, perhaps grabbing a couple pertinent OSGi plugins and going from there. However, from what I can tell, all the involved plugins are intricately tied into many layers of Designer-the-IDE and so are no small matter to use on their own without also including the entire stack. So that left me to cobble together an equivalent process out of parts.

Fortunately, a couple projects have already provided a solid foundation for this. First and foremost is the XPages Bazaar. This is a project that Philippe Riand created a number of years ago, meant to be a workshop for really experimental components in a form less contrained than the ExtLib became. Since he left IBM, it's sat unmaintained, but I figured it'd be a perfect incubator for this project, so I tossed it up on GitHub, recomposed its Maven structure, and cleaned it up a bit for FP10 use. The reason why it makes such a perfect shell is a pair of its features: an XSP interpreter and an on-the-fly Java compiler. The former hooks into the mysterious guts of the XSP runtime to allow for translation of XSP to Java and the latter wraps the official Java compiler API with some OSGi knowledge to compile that along into bytecode.

Even starting with this, the first couple steps still required a lot of digging around. I learned how to install and activate OSGi bundles, how XPages Registries work internally, and made some tweaks to work around problems I encountered. I also encountered the joy of a bizarre javac bug to do with annotations in enum constructors, which my target project used.

Once I had the XPages-side components compiled, the next step was to start composing the NSF. The ODP format for XPages elements and other "file resource"-type entities is to put the code in its "normal" form in a file and then a subset of the DXL in a ".metadata" file next to it. The trouble here is that, even for entities where the file data is stored in the note unprocessed, the storage format isn't a strict binary blob of the file data: it's a composite data stream of file header and segment structures. I thought of two main ways I could go about getting these file resources into the NSF: via IBM's NAPI and by building the structures into the DXL files before import. The NAPI has a convenient FileAccess class for this purpose (presumably used by Designer), but my attempts to use it met primarily with server crashes. I'm sure it's possible to go this route, but I'd already "solved" the DXL problem years ago, for ODA's Design API. So, at least for now, I took the tack of writing out the binary structure manually, pouring it into the DXL as Base64, and importing that. It's a little inefficient, but it works.

Overall, I've made a lot of progress so far, but there's still a lot to be done: not all file types have their data put into the right places, LotusScript isn't properly compiled, Agents don't do anything at all yet, and XPages+CCs aren't actually imported into the NSF. Still, it's in a spot where I'm confident that it can one day work, whichis more than I could have said a week ago. If you'd like, browse around the code and pitch in if it's an itch you'd like to scratch as well.

NSF ODP Tooling 1.0

  • Apr 28, 2018

A couple weeks back, I started a new project, and today I decided to declare it a 1.0. The premise of this project is simple: I really, really hate Designer. Since the original post, it's expanded into a set of tools that cover three main tasks:

ODP Compilation

The main impetus of the whole thing was to get a way to compile On-Disk Projects into working NSFs without using the extremely-fiddly Headless Designer route (and thus also decoupling the build process from Windows). In 1.0, ODP compilation works by installing a set of plugins on a Domino server (Windows or Linux), configuring some Maven properties, and wrapping the ODP in a Maven project. That process can upload dependent OSGi plugins and compile complex XPages apps as well as import the expected normal legacy Notes resources.

NSF Deployment

In addition to compilation, the Maven wrapper can be configured to deploy the NSF to a Domino server also running the plugins. This portion isn't as battle-hardened as compilation, but it seems to work, as long as your server ID is authorized to run the resultant XPages application, since that's how it'll be signed. Perhaps in the future I'll add the ability to sign with an ID out of an ID Vault.

Eclipse Tooling

In Eclipse Neon or above, the tooling adds a bit of knowledge about how to handle ODP Maven projects as well as some autocomplete capabilities for editing XSP source. Currently, autocomplete knows about the components that come with Domino 9.0.1 FP 10 as well as any Custom Controls inside the NSF, but in time I'd also like to include XPages-Library-contributed components. Additionally, it configures the project as a Plug-in Project with dependencies based on the selected XPages libraries and with source folders and embedded jars set up in the classpath.

The End Result

This project started out as just wanting to get Jenkins compilation working more reliably, but it's grown a bit to also allow for a smoother developer experience when working with NSF data in Eclipse. The "use case", for lack of a better term, that I'm aiming for is when you have the bulk of your code in OSGi plugins but have an NSF to maintain. I don't have any interest in replacing every editor from Designer, but I'd like to have enough that it's practical to do some basic XPages and legacy dev work without having to go over to Designer for everything. It's not fully there yet, but this version is a good start.

Getting The Code

The project is hosted on OpenNTF, and the source code is hosted on GitHub.

NSF ODP Tooling Example Project

  • Apr 29, 2018

To go alongside the first proper release of my NSF ODP Tooling, I've added an example project to the Git repository:

https://github.com/OpenNTF/org.openntf.nsfodp/tree/master/example

This project demonstrates how to use the tooling to create an XPages library project and build an NSF that uses it within the same Maven tree. This example project also serves as a reasonable template for the standard kind of project setup I make for Domino nowadays, minus a compile-time test plugin (which I'll probably add in eventually).

Environment Setup

Before building the ODP, you'll need to set up a compilation server and configure Maven to know about it. To start out with, make sure you have a Notes-ified Maven environment as described here. Since the IBM-provided update site is quite old at this point, it may be worth updating it from your local installation.

Next, install the Domino plugins on a Domino server running at least 9.0.1 FP10. It's best to do this on a pristine server without non-standard plugins installed, since part of the compilation process is to load and unload the needed plugins for your project. For my needs, I set up a Linux VM and it's doing the job nicely. Once that's set up, configure your Maven settings.xml to reference your compiler server, merging these values in with the normal Notes properties:

<?xml version="1.0"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
    <profiles>
        <profile>
            <id>main</id>
            <properties>
                <notes-platform>file:///Users/jesse/Documents/Java/IBM/Notes9.0.1fp10</notes-platform>
                <nsfodp.compiler.server>someserver</nsfodp.compiler.server>
                <nsfodp.compiler.serverUrl>http://some.server/</nsfodp.compiler.serverUrl>
            </properties>
        </profile>
    </profiles>
    <activeProfiles>
        <activeProfile>main</activeProfile>
    </activeProfiles>
    <servers>
        <server>
            <id>someserver</id>
            <username>builduser</username>
            <password>buildpassword</password>
        </server>
    </servers>
</settings>

Note the "server" block at the bottom to provide login credentials. The plugins require a non-anymous user - at the moment, they allow ANY non-anonymous user, though, so you can create a new user just for compilation purposes. It's also good practice to encrypt your server connection passwords.

There are also options for deploying the NSF, but, since that's less important for our needs, I'll leave that aside for now. The README has a bit more information about that.

The Example Project

The structure of the projects is similar to the one I detailed in my Java series a couple of years back, but it's evolved a bit since then. The main aspects of it of note are:

The Folder Structure

Lately, I've been following the advice from the Vogella blog about how to structure an OSGi project. In a case like this, where there is only one plugin, one feature, and one update site, it's overkill, but I've found it to be better to create this structure at the start so you don't end up with a big mess of projects serving different needs within the same flat folder.

The nsfodp-maven-plugin

The most pertinent addition is the ODP wrapped in a Maven project. In the nsfs/nsf-example folder, I created a pom.xml to configure a project of type domino-nsf, which expects by default the ODP to be in an odp folder immediately within it. The ODP in there is entirely normal: it's just a near-fresh NSF exported from Designer, with the main addition being the inclusion of the XPages Library.

The project's pom does a few things: it establishes it as being a domino-nsf project and it adds some additional configuration to the nsfodp-maven-plugin, telling it to include the update site generated earlier in the build as part of the compilation process; this is what allows the NSF to build with the XPages library in the current project.

License Management

Since I've been making contributions for OpenNTF and particularly since taking over as IP manager, I've developed a much better appreciation for dotting my is and crossing my ts when it comes to licensing. One of the tools I quite like for this is the license-maven-plugin from com.mycila (it's not the only one of that name, and any of them should do the job). This plugin allows you to specify a license header template to be added to the top of each of your source files, which is an otherwise-tedious task that's easy to neglect. Once I added that to the root pom and set up appropriate exclusions (definitely make sure to add exclusions for any third-party code!), I'm able to run mvn license:format from the root project and have it run through all the source files in the directory tree and add an appropriately-formatted license header. I've definitely made this a standard part of my project setup now.

Plugin Versioning and maven-enforcer-plugin

This one admittedly doesn't have that much impact on day-to-day development, but it's another good "keeping your ducks in a row" addition: I've taken to explicitly specifying the versions for my Maven plugins, even when the version is implied by the core Maven version. This "locking down" can reduce one cause of mysterious code breaking if an implicit inclusion is upgraded in a way that is incompatible with your build process.

My friend in this task is the versions-maven-plugin's versions:display-plugin-updates goal, which will look through your project tree and find plugins that have newer versions in the available repositories and also tell you what your implied plugin versions are from the super-pom. I use this information to explicitly enumerate the plugins and find updates - the "copy and paste this block of XML" nature of Maven means that it's very easy to end up running a plugin that's several major versions behind.

Alongside this, the goal will tell you what the minimum required Maven version is for everything you're doing, which I've taken to specifying in the old-style prerequisites block as well as via the newer-style maven-enforcer-plugin route. Laying out these requirements explicitly is another good way to avoid phantom problems. Note that, for Eclipse, it's good to add a m2e configuration block to ignore the enforcer line, since m2e doesn't know what to do with it.

The OpenNTF Artifactory Server

This isn't so much a new technique as it is a prerequisite for using the plugin: currently, the plugin is hosted on OpenNTF's Artifactory server and not Maven Central, so you'll have to add a pluginRepository block for it. Once you have that, you can add the nsfodp-maven-plugin to the root pom.

The Eclipse Tooling

Once you have a project configured in Maven, the next step is to install the Eclipse tooling. This can be installed into any Eclipse installation running Neon or newer in a Java 8+ JVM. For my use nowadays, I primarily use Oxygen.3a on the Mac, but any platform should work.

Once you have that installed, you can import the projects into your Eclipse workspace and the tooling will adapt some elements for use as a pseudo plugin project. It will auto-generate MANIFEST.MF and build.properties files at the project's top level (which is why it's important to have the ODP in a sub-directory, so the FP10+ MANIFEST.MF isn't overwritten) and use those to configure the XPages Java class path, requiring the same plugins as the NSF does, including XPages library plugins, as well as adding any used Jars to the classpath. The result is that you can use it to edit your Java code with full classpath knowledge:

Eclipse Project

Beyond that, it provides some autocomplete capabilities for editing .xsp files. Currently, it had built-in knowledge of the controls that come standard on a Domoino 9.0.1FP10 server, plus any custom controls in your application. This autocomplete takes the form of contributions to Eclipse's standard XML editor, so it's pretty snappy:

Autocomplete

This works for tag names as well as component properties.

Limitations

This tooling has some significant limitations:

  • The biggest is that it doesn't have any special knowledge of most design elements - and, if you use binary DXL for safety purposes, that means that most legacy elements are difficult to modify.
  • It doesn't currently do any programmatic pairing between editable design elements and their associated .metadata and .xsp-config files, so it's best to do keep Designer around for creating those.
  • The XSP autocomplete consists just of contributions to the autocomplete list, and so it doesn't do any checking for legality of content or tag placement, nor does it currently have any descriptive metadata.

Future Prospects

I've gotten this project to a point where I can reduce my level of daily annoyance with my tools, which is an important step. There are a few more things that I'd like to add so I can further reduce my need to use Designer at all: giving the editors some knowledge of split files could allow for manipulating custom control properties in a better way, some property panes may be worth making, I'd like to have a way to modify binary-format DXL notes (which may either be by using ODA's CD structure implementation or by round-tripping the DXL to Domino to convert it to friendly format for editing), and I'd like to eliminate the strict requirement of having a Domino server around for compilation.

The last one is a fun project on its own: my plan is to have a headless Java app loading up an Equinox environment and using the same plugins and REST services as the Domino server. That's mostly functional now, but it has some odd compilation-time bugs that will take some investigation. With that in place, it'd remove the need for any separate server, though it would then require that your development environment have a Notes or Domino runtime available.

As always, I'd welcome any contributions, especially if someone has a particular itch they'd like to scratch. I have some open issues in the GitHub project that I likely want to tackle at some point, and I'm sure this could cover a lot more ground besides.

NSF ODP Tooling 1.2

  • Jun 12, 2018

I've just published a new release of the NSF ODP Tooling, and this one is important by virtue of the fact that it covers enough bases for me to put it into production with my largest active XPages project.

Since the 1.0 release, I've added a couple important new Maven plugin options in addition to general bug fixes:

  • "compilerLevel": by default, it compiles to the Domino server's Java version (currently 1.8 in the minimum required configuration), but now it is possible to specify 1.6 to target older Domino releases for production
  • "appendTimestampToTitle": append a timestamp to the database title during compilation, which is useful to see when going to deploy the NTFs to production
  • "templateName": set a name to be used in the $TemplateBuild shared field, which is a nice bit of fit and finish when making a template. This also sets the version (based on the Maven project version) and build date fields
  • "setProductionXspOptions": to enable compressed JS and resource aggregation in the compiled NSF, useful to use the inefficient options for development/debugging but get better performance in deployment

I've also gradually improved the Eclipse side, though that can use a lot more work. Just having the in-NSF Java classpath working is a huge boon for development and refactoring, and it'd be great to eventually have tooling available to create and edit design elements with some proper knowledge of how they work, to keep the metadata in sync.

As it is, this project has been tremendously useful for me so far, easing a big burden - I can't tell you how much time I've lost switching branches between a release candidate and develop, trying to coax Designer into properly picking up the changed files and recompiling, and then prepping the NSFs for deployment (even with tools to aid me). With the ODPs wrapped in a Maven tree, I have Jenkins take care of all of that for me, and more reliably to boot.