Showing posts for tag "domino"

The Intricate Work of OSGi Dependencies on Domino

Dec 2, 2020, 3:03 PM

Tags: domino osgi
  1. Converting Tycho Projects to maven-bundle-plugin, Initial Phase
  2. Winter Project #2: Maven P2 Repository Resolver
  3. OpenNTF Fork of p2-maven-plugin
  4. The Intricate Work of OSGi Dependencies on Domino

One of the main goals of OSGi is proper runtime dependency management, not only allowing a bundle to declare what its dependencies are to ensure that they're there, but even to select one of multiple available versions. For example, you might have one bundle that expects Guava 15 but not higher and another that expects 18 or above, and both can be loaded successfully if you have multiple Guava versions installed. The goal is that you're supposed to be able to just throw a whole bunch of bundles into a pot and the system will figure it out.

Before I get into why this system is hobbled on Domino, I'll add a little more background.


For our purposes here, bundles have two main ways to declare what they are (there are more than this, but they're not relevant right now):

  • The bundle's symbolic name combined with its bundle version. For example, the core bundle for ODA is named "org.openntf.domino" and has a version like "".
  • The bundle's exported packages, which are Java class packages and might also individually have versions. Using ODA again as an example, it exports a slew of packages, such as org.openntf.domino and org.openntf.domino.nsfdata, though it doesn't specify versions for any of these.

The versions of the bundle and the exported packages don't need to be the same, nor do all the exported packages need to have the same version. This shows up a lot in "spec bundles", where a vendor will wrap a standard spec API in their own bundle for various reasons. For example, Apache Geronimo has a bundle called "org.apache.geronimo.specs.geronimo-jaxrs_2.1_spec", which provides the JAX-RS 2.1 spec. The bundle itself has a version of "1.1.0", while the exported packages are all version "2.1".

To require a bundle by name, you use the Require-Bundle header, like Require-Bundle: org.openntf.domino;bundle-version="11.0.0", which requires specifically the ODA bundle, version 11 or above. To require a package, you use Import-Package, like Import-Package:;version="2.1.0". The two methods have some implications when it comes to how the ClassLoaders work, which I touched on a bit earlier this year.

Culture Differences

The package-based mechanism is generally preferred nowadays over the bundle-based one, largely for the kind of flexibility and division-of-responsibilities it provides. For example, if I want version 2.1.0 of, my code shouldn't care at all whether it comes from Geronimo's bundle, JBoss's, or anywhere else - it's all the same thing (in theory). This is extremely common for projects created with bnd and tools based on it, which can generated imported and exported packages automatically based on your code and some small configuration. For example, the bundles that make up Open Liberty use bnd config files that have some loose configuration, which then is processed out to full listings of packages with version information. This is also often paired with OSGi's "Capability" system, but that's one of the "not important for now" things.

Domino - and I believe this is inherited from Eclipse, which does the same thing - is largely based on bundle requirements. For example, the org.eclipse.jdt.ui bundle (part of the Java development tools in Eclipse and Designer) requires a bevy of bundles by name version and doesn't tag any of its exported packages with versions. This is similarly reinforced in the tools. Eclipse, unsurprisingly, uses Tycho to bring the "Eclipse PDE" style to the table, where the MANIFEST.MF file is more hand-crafted, and the tools don't do as much for you automatically. You can go full Import-Package and attach versions to your exported packages with Eclipse, but the tooling doesn't encourage it.

Conflicts in Practice

That brings us back to how this all contributes to making working with OSGi a little extra annoying on Domino. This is something that comes up constantly in my XPages Jakarta EE Support project: I want to bring in an implementation component for a JEE spec, but it will either already have or will have generated for it OSGi rules that lean towards the "package-style" of doing things, versions and all. Because there are common packages (like, say, javax.activation) that are supplied by bundles present in the Domino runtime, I want to use those. However, since the packages from Domino don't have versions specified, I need to re-wrap the bundles to import the packages without versions. Thus begins this ongoing nightmare.

Another approach I could take would be to bring my own, nicely-OSGi-ified versions of the afflicted packages, and options abound. However, that leads to a sneaky other trouble: because some of the Domino bundles are multi-spec monsters - with being the absolute worst culprit - some other bundle on the system might casually import javax.activation and javax.mail. If I have this other spec implementation floating around, then it could get matched to my bundle for the former and IBM's for the latter... and crash right into "exposed to a package via two dependency chains" problem. That wouldn't necessarily be an issue if everything used versions on packages, but leaving them off means it's kind of up to the container to match bundles to each other, and it's entirely happy creating impossible conflicts.

Ways Around It

When working through this sort of trouble, I've found a few ways around the things I've run into. The first is what I mentioned above: using p2-maven-plugin to re-wrap bundles with instructions that make them more Domino-friendly. This involves a few tricks:

Aside from reworking existing bundles, I have a few times ended up creating fragment bundles that glom onto one bundle to tie it to another one after the fact, usually for the components that bridge different JEE specs.

The Wildcard: The System Bundle

In general, with OSGi, you can expect the packages you need to be provided in bundles, but it also allows for the core JDK to be implicitly available - that is, things like java.lang don't need to be imported, and are always present. That's fine, since it's generally well-enough-defined what makes up the JDK and what you'd need to instead bring in.

However, the Domino core classpath doesn't contain just the JDK, but also anything in jvm/lib/ext and (as a curveball) ndext from the Domino directory. Above, I specifically pointed out the javax.activation package, and that's because it suffers the most from both the javamail bundle as well as this. If you run tell http osgi packages javax.activation on Domino's console, you should see that bundles get this from two places: some use, while others use the system bundle, org.eclipse.osgi. That "system bundle" concept is not only the thing in charge, but it also passively provides access to classes found in the lower-level classpath. In a fully OSGi environment, that system classpath will be pretty clean, but Domino's isn't that.

The good news here is that it isn't usually a big problem. If you import packages by a non-zero version, they'll never match from the system bundle this way. Still, it's always there, lurking, and the problems increase if you start adding more JAR files to jvm/lib/ext to avoid policy or amgr-memory issues. We've seen this periodically with ODA, where we used to support the use case of putting the core files there and using just a shim at the XSP layer, before it became too much of a hassle to manage. I ran into it again more recently when Guava showed up there: because I hadn't specified a version, it was able to match from the system bundle, and ended up with classes available at compile time but missing at runtime.

The Upshot

The upshot here is that there's no simple advice for dealing with this. Cultural and implementation factors make bringing third-party code into Domino unusually difficult, but it can generally be dealt with via various patching mechanisms. The XPages JEE project's dependency module ended up turning into a trove of such workarounds, so perhaps it can be useful if you ever run into this sort of thing yourself.

Writing Domino Server Addins With GraalVM Native Image

Sep 27, 2020, 3:35 PM

Tags: graalvm domino

I was thinking the other day about the task of writing a Domino server addin, the kind that you run by typing load foo on the server console. The way this is generally done is via C or the like: you write a program using your dusty old copy of the C API Toolkit and have an AddinMain function as the entrypoint. That's fine enough if you want to write in C, but, even beyond the language, it carries the tremendous overhead of a fiddly compilation chain that differs per-platform.

I got to thinking, then, about GraalVM, and specifically its Native Image capability. Before I get into what I did, I figure this warrants some background.

What is GraalVM?

GraalVM is a project from Oracle that is, roughly, an alternative core Java Virtual Machine. It's designed to serve a number of goals, but the main ways that I've seen it used is to improve the speed and efficiency of Java-based programs. It also has some neat-looking capabilities for running multiple languages in one app space, but I have yet to look into that.

The Native Image capability is a way to compile Java applications to native executables for a given platform. So, instead of having a JAR file that you then run with an installed JVM, you'd have an executable that you run directly, and which effectively acts as its own "VM". This means you end up with just "some executable" on your system, and the lack of bootstrapping needed to run it opens up some possibilities.

Domino Server Addins

Though Domino server addins have their own set of functions within the Notes C API, they're really just an executable that Domino launches as a sub-process. If you have a basic executable named foo in your Domino program directory, you can type load foo and it'll run it, whether or not the executable does anything with the Notes API at all. It won't necessarily be useful if it doesn't use the Notes API, but it'll run.

It's this "just an executable" bit, though, that was a contributing factor to making Java not a practical language for this. That's also where RunJava fit in: the runjava executable just initialized a JVM and loads the named class, which is afterward responsible for everything, but that was nonetheless obligatory work to get a Java app loaded this way.

The Combination

Once I realized these things, it wasn't a far reach to try implementing an addin this way. One of my initial concerns was the way addins use AddinMain as a C-type entrypoint - my knowledge of how that sort of thing works is limited enough that I wasn't sure if GraalVM's annotations would suffice. However, the C API documentation relieved my worry: using that function name is just a convenience that handles some of the bootstrapping for you. If you just use a normal main(...) entrypoint, the only difference is that you're on the hook for managing your status line more (the thing that shows up when you do show tasks).

Fortunately, the addin-related methods in the lotus.notes.addin.JavaServerAddin class in Notes.jar are extremely-thin wrappers around native calls and aren't actually specific to RunJava in any way. You can subclass it and use it in essentially the same way as in a RunJava addin:

package frostillicus.graalvm;

import lotus.domino.NotesException;
import lotus.notes.addins.JavaServerAddin;

public class Main extends JavaServerAddin {
	static {
		System.setProperty("java.library.path", "/opt/hcl/domino/notes/11000100/linux"); //$NON-NLS-1$ //$NON-NLS-2$
		System.loadLibrary("notes"); //$NON-NLS-1$
		System.loadLibrary("lsxbe"); //$NON-NLS-1$
	public static void main(String[] args) {
		new Main().start();
	public Main() {
		setName("GraalVM Test");
	public void runNotes() throws NotesException {
		AddInLogMessageText("GraalVM Test initialized");
		int taskId = AddInCreateStatusLine(getName());
		try {

			// Do your work here

		} catch(Throwable t) {
		} finally {


GraalVM-specific configuration

The GraalVM project provides a Maven plugin to do native compilation for you, and I make use of that in the project's pom.xml:

		<!-- snip <buildArgs> -->

Including that in your project will produce a native executable for your current platform in the target folder, alongside the normal JAR file.

The bit I snipped out, though, ends up being important. In a similar way to what happens during Android "Java" compilation, the GraalVM native compiler builds a map of all of the code used in your project to create its native representation. Additionally, it doesn't support reflection as casually as a normal JVM does, and doing a compilation like this shows just how common reflection is in Java.

Reflection and JNI Configuration

What reflection (and JNI) in Java generally needs is a mapping table of class/method/field names to their class representations, and GraalVM doesn't build this for everything by default. Instead, it does its best guess based on your actual code, but then it's up to you to explicitly specify the parts you'll be accessing dynamically.

For the normal case, Oracle wrote a tool that will monitor an actively-running app in Java for such calls. You build your app and run it non-native with this agent, and then it will spit out a configuration file based on the actually-called reflective methods.

However, as with everything else to do with Domino, it's not the normal case: since what I'm running only reasonably exists when launched explicitly from a server, I had to do it the "hard" way. Fortunately, the it's actually just mostly tedious: build the app, launch the Domino Docker container, wait to look for a NoClassDefFoundError or related problem, add that to the config file, and repeat until it stops yelling. Some cases are a little fiddlier, like how JNA's native component misrepresents the class name it was trying to find, but overall it's just time-consuming.


So, this is possible, but is it worth doing? Depending on what you want to do, maybe. It's mildly less unsupported than RunJava, and has the huge advantage of not polluting the server's classpath with all of your application code. Additionally, it should be pretty zippy, as GraalVM boasts some impressive performance numbers. Additionally, at least for Java developers, it's much, much easier to use the native-image-maven-plugin than it is to set up cmake or manual makefiles for a C/etc. project.

However, it can also be a real PITA to get working, especially for a reflection-heavy project. Additionally, though you're technically using Addin* functions with a native executable, it's not like HCL would take your call if you run into trouble with a monstrosity like this (I assume). Most importantly, it's restricted to the sort of thing that would make sense as a server addin to begin with - for example, this wouldn't help with building web apps unless you were planning to use it to (again, just as an example) run a web server that's written in Java.

Future Tinkering

I think that this warrants some more investigation. I'd be curious if this process would work for writing other native components, such as DSAPI filters and ExtMgr addins. In those cases, it absolutely would be important to have the right entrypoints, so it wouldn't be quite so easy. Still, it'd be neat if that worked.

And GraalVM and the Native Image component are definitely worth some time even aside from anything Domino-related. I'm curious about what you can do with the "polyglot" features, for example.

Example Project

I've put an example project up on GitHub, which is a basic example that just accepts strings via tell graalvm-test foo and echoes them back. It also includes a Dockerfile for running via HCL's official Domino 11.0.1 image. I haven't actually tested it any other way, so that's the best way to give it a shot.

Weekend Domino-Apps-in-Docker Experimentation

Jun 28, 2020, 6:37 PM

  1. Weekend Domino-Apps-in-Docker Experimentation
  2. Executing a Complicated OSGi-NSF-Surefire-NPM Build With Docker
  3. Getting to Appreciate the Idioms of Docker

For a couple of years now, first IBM and then HCL have worked on and adapted community work to get Domino running in Docker. I've observed this for a while, but haven't had a particular need: while it's nice and all to be able to spin up a Domino server in Docker, it's primarily an "admin" thing. I have my suite of development Domino servers in VMs, and they're chugging along fine.

However, a thought has always gnawed at the back of my mind: a big pitch of Docker is that it makes not just deployment consistent, but also development, taking away a chunk of the hassle of setting up all sorts of associated tools around development. It's never been difficult, per se, to install a Postgres server, but it's all the better to be able to just say that your app expects to have one around and let the tooling handle the specifics for you. Domino isn't quite as Docker-friendly as Postgres or other tools, but the work done to get the official image going with 11.0.1 brought it closer to practicality. This weekend, I figured I'd give it a shot.

The Problem

It's worth taking a moment to explain why it'd be worth bothering with this sort of setup at all. The core trouble is that running an app with a Notes runtime is extremely annoying. You have to make sure that you're pointing at the right libraries, they're all in the right place to be available in their internal dependency tree, you have to set a bunch of environment variables, and you have to make sure that you provide specialized contextual info, like an ID file. You actually have the easiest time on Windows, though it's still a bit of a hurdle. Linux and macOS have their own impediments, though, some of which can be showstoppers for certain tasks. They're impediments worth overcoming to avoid having to use Windows, but they're impediments nonetheless.

The Setup

But back to Docker.

For a little while now, the Eclipse Marketplace has had a prominent spot for Codewind, an IBM-led Eclipse Foundation project to improve the experience of development with Docker containers. The project supplies plugins for Eclipse, IntelliJ, and VS Code / Eclipse Che, but I still spend most of my time in Eclipse, so I went with the former.

To begin with, I started with the default "Open Liberty" project you get when you create a new project with the tooling. As I looked at it, I realized with a bit of relief that there's not too much special about the project itself: it's a normal Maven project with war packaging that brings in some common dependencies. There's no Maven build step that expects Docker at all. The specialized behavior comes (unsurprisingly, if you use Docker already) in the Dockerfile, which goes through the process of building the app, extracting the important build results into a container based on the open-liberty runtime image, bringing in support files from the project, and launching Liberty. Nothing crazy, and the vast majority of the code more shows off MicroProfile features than anything about Docker specifically.

Bringing in Domino

The Docker image that HCL provides is a fully-fledged server, but I don't really care about that: all I really need is the sweet, sweet and associated support libraries. Still, the easiest way to accomplish that is to just copy in the whole /opt/hcl/domino/notes/11000100/linux directory. It's a little wasteful, and I plan to find just what's needed later, but it works to do that.

Once you have that, you need to do the "user side" of it: the ID file and configuration. With a fully-installed Domino server, the data directory balloons in side rapidly, but you don't actually need the vast majority of it if you just want to use the runtime. In fact, all you really need is an ID file, a notes.ini, and a names.nsf - and the latter two can even be massively trimmed down. They do need to be custom for your environment, unfortunately, but at least it's much easier to provide just a few files than spin up and maintain a whole server or run the Notes client locally.

Then, after you've extracted the juicy innards of the Domino image and provided your local resources, you can call NotesInitExtended pointing to your data directory (/local/notesdata in the HCL Docker image convention) and the notes.ini, and voila: you have a running app that can make local and remote Notes native API calls.

Example Project

I uploaded a tiny project to demonstrate this to GitHub: All it does is provide one JAX-RS resource that emits the server ID, but that shows the Notes API working. In this case, I used the Darwino Domino NAPI (which I really need to refresh from upstream), but Domino JNA would also work. Notes.jar would too, but I think you'll need one of those projects to do the NotesInitExtended call with arguments.

The Dockerfile for the project goes through the steps enumerated above, based on how the original example image does it, and was tweaked to bring in the Domino runtime and support files. I stripped the Liberty-specific stuff out of the pom.xml - I think that the original route the example did of packaging up the whole server and then pulling it apart in Docker image creation has its uses, but isn't needed here.

Much like the pom.xml, the code itself is slim and doesn't explicitly refer to Docker at all. I have a ServletContextListener to init and term the Notes runtime, as well as a Filter implementation to init/term the request thread, but otherwise it just calls the Notes API with no fuss.

Larger Projects

I haven't yet tried this with larger projects, but there's no reason it shouldn't work. The build-deploy-run cycle takes a bit more time with Docker than with just a Liberty server embedded in Eclipse normally, but the consistency may be worth it. I've gotten used to running a killall -KILL java whenever an errant process gloms on to my Notes ID file and causes the server to stop being able to init the runtime, but I'd be glad to be done with that forever. And, for my largest project - the one with the hundreds of XPages and CCs - I don't see why that wouldn't work here too.

Normal Domino Projects

Another route that I've considered for Domino in Docker is to use it to deploy NSFs and OSGi projects. This would involve using the Domino image for its intended purpose of running a full server, but configuring the INI to just serve HTTP, and having the Dockerfile place the built OSGi plugins and NSFs in their right places. This would certainly be much faster than the build-deploy-run cycle of replacing NSF designs and deploying the plugins to an Update Site NSF, though there would be a few hurdles to get over. Not impossible, though.

I figure I'll kick the tires on this some more this week - maybe try deploying the aforementioned giant XPages .war project to it - to see if it will fit into my workflow. There's a chance that the increased deployment times won't be worth it, and I won't really gain the "consistent with production" advantages of Docker when the way I'm developing the app is already a wildly-unsupported configuration. It might be worth it if I try the remote mode of Codewind, though: I have some Liberty servers that Jenkins deploys to, but it'd be even-better to be able to show my running app to co-developers to work on something immediately, instead of waiting for the full build. It's worth some investigation, anyway.

The Lay of the Java Land, Early 2020

Mar 4, 2020, 2:34 PM

Tags: domino

I was musing earlier (somewhat incorrectly) about the weird state of Java versions, and it got me thinking about how odd the landscape looks in general lately, even as things are progressing splendidly. And, since not everyone follows a dozen Java-related Twitter feeds to keep up on all this stuff - and because Domino has ignored so much of this for so long - I figured it'd be useful to have a summary.

Java Itself

For starters, there's Java-the-language, which has gone through some changes both in how it's licensed and how it evolves recently, largely for the better.

Release Cadence

The most notable change is how the version pace has picked up. Originally, Java was updated pretty regularly, but things slowed down after Java 6's release in late 2006. For whatever reasons, Sun took a IE-6-style break for a while and didn't come out with 7 until 2011 and then 8 in 2014. Starting with 9 in September 2017, though, some big changes came in:

  • New integer Java releases come out every six months
  • Starting with 11, "long-term support" releases will come out every three years, or every six integer versions

So Java 10 came out in March 2019 and 11 came out in September 2018, but 11 is the next "real" release after 8. Similarly, 12 and 13 have come out in the intervening time, but it won't be until Java 17 in September 2021 that the next LTS will arrive. Similar to other rapid+LTS release systems like Ubuntu's, the "intermediate" releases are expected to be stable and ready for production, but you can only expect to get commercial support for the current intermediate release plus any LTS releases still in their support window.

Along those lines, one could reasonably expect Java server vendors to stick to the LTS releases, even ones like HCL that aren't licensing supported builds from IBM or Oracle.

Preview Features

One of the main points of the faster release cadence is to allow quicker delivery of new features, and paired with that is Oracle's new willingness to put "preview" features in a release before they can call them officially solid and giving them cover if they decide they want to break the syntax. These are enabled with special flags at compile- and run-time, and currently include a handful of nifty features like yield in switch blocks and multi-line strings (a type of "heredocs") that will almost definitely become "official" features by the time the next LTS rolls around.

Removal of Features

They also gave themselves permission to remove features from the core JRE distribution, in particular a handful of specs that have gotten less used over time or otherwise make more sense to be part of Java/Jakarta EE specifically. Of note for Domino developers is the CORBA API, which we almost never use directly but which is required to load Notes.jar. That's gone in Java 11, and so (for now at least) using Notes.jar on Java 11+ requires including a replacement implementation.


I mentioned it in a Java grab bag before, but it's worth reiterating: Java (the language and the platform) is free and open-source for all to use, but Oracle's specific JDK/JRE builds are not. For all intents and purposes, AdoptOpenJDK is the sole go-to place to get Java for private and commercial use now, unless you specifically want to pay Oracle or IBM for support.

Additionally, there are now two core JVMs to choose from when downloading Java: HotSpot and OpenJ9. HotSpot is essentially the "normal" one, the Java core that Sun/Oracle have been shipping forever. OpenJ9 is an open-source version of the J9 JVM that IBM has long maintained and put into all of their products (Domino included. IBM open-sourced it and contributed it to the Eclipse Foundation (they'll come up again shortly), and it's making a name for itself as a solid lean and quick-to-start runtime for containerized systems in particular.


The world of Java server technology, separate from the language has also been going through some churn with positive results. In this case, I mostly mean Java/Jakarta EE - Spring has been chugging along without too much apparent turmoil. Additionally, I don't know too much about Spring, so I won't cover it here.

Over the last decade, Oracle seemed to generally lose a lot of interest in developing Java EE - while they still put out new releases, they started getting further apart and the overall sense was that Oracle would be much happier if they just didn't have to deal with it anymore.


It was around this time that the community outside of Oracle's JEE team got a little antsy about this slowdown and lack of focus and started the Eclipse Microprofile project. It started out as a thin subset of JEE technologies - just JAX-RS, JSON-P, and CDI - tailored for the purposes of making it easier to write microservices with an extremely-low footprint. Quickly, though, it grew beyond just a selection of existing specs and started to grow new specifications that JEE didn't have to support the mission. Beyond just microservices-specific improvements, these specs also bring along tools that are handy in any old application, like an improved REST client and an annotation-based Configuration API. Microprofile turned into the go-to place for new development in the JEE world while Oracle was slowing down.

Jakarta EE

Oracle managed to get the (splendid) Java EE 8 release out the door eventually, but decided they had had enough of shepherding the platform. Fortunately, instead of consigning it to a slow death, they handed the reigns over to Eclipse, which formed the awkwardly-named EE4J project to oversee it. Since Oracle didn't give them the rights to use the name "Java", the actual platform itself was rebranded as "Jakarta EE", which had its own "Eclipse-washed" Jakarta EE 8 release.

That transition has gone very well, though there's one hurdle on the horizon: Oracle also didn't grant the rights to use the javax.* namespace for any new specifications, and so Eclipse decided to make the jump in Jakarta EE 9 to switch all of the specifications over to jakarta.*. What this means in practice is that all of those javax classes (like Servlet) we use now will, in their JEE 9 incarnations, be renamed in the style of jakarta.servlet.Servlet. There will likely be tools in various IDEs and toolchains to help with the conversion, and I suspect that app servers like Liberty will still support the old names for a good while, but it'll be a weird time.


It's also a bit of a weird time for the pairing of Jakarta EE and Microprofile. The latter came about when the former was moribund, but now they're both active and within the same open-source organization. Microprofile's remit isn't exactly the same as Jakarta's, so they're both going to continue as-is for at least a good while. Still, it feels a bit odd to have two Java server frameworks in the same place, and so it's possible that Microprofile will either be subsumed into JEE (like how JEE already has "web" and "full" profiles) or will be something of an innovation area to push new technology faster before sending it back upstream to JEE. That'll be interesting to watch.

GraalVM and Quarkus

Finally, I'll mention a couple "miscellaneous" technologies that have been developing in the background while all of this was happening.

GraalVM (which is presumably like the French word for "grail" and not pronounced like something a goblin would say) is a variant of the JVM core that Oracle has been working on for a few years. I think it's meant to be roughly equivalent to LLVM but for the Java world: higher performance than before, multi-language support, and compilation down to native binaries. It's an open-source (GPL) project and, while Oracle offers a paid Enterprise Edition, the Community Edition is legal for production use.

Alongside this has come along Quarkus, which is a Microprofile implementation laser-focused on speed and resource usage. It doesn't require GraalVM's native compilation, but the pairing of the two is a prime part of Quarkus's message. Quarkus isn't a full Jakarta EE server, but it's a very-intriguing purpose-built stack for developing speedy Java server apps primarily for containers. It's also a good conceptual example of allowing developers to write fairly-dynamic code (like CDI injection) but then turning that into concrete bindings at compilation time instead of deferring all lookups to runtime.

It's been on my list to kick the tires on these things specifically for a little while now. They're both hitting their stride lately, so I suspect that they'll have interesting effects over time.

Writing Domino JEE Apps Outside Domino

Nov 8, 2019, 1:47 PM

In my last post, I mentioned that I decided to write the File Store app as a Jakarta EE application running in a web server alongside Domino, instead of as an app running on the server itself. In the comments, Fredrik Norling asked the natural question of whether it'd have been difficult to run this on the server, which in turn implies a lot of questions about deployment, toolkits, and other aspects.

Why Not

My immediate answer is that it would certainly be doable to run this on Domino, but the targeted nature of the app meant that I had some leeway in how I structured it. There are a couple reasons why I went this route, but most of them just boil down to not having to deal with all the gotchas, big and small, of doing development on top of Domino.

As a minor example, I wanted to add some configuration parameters to the app, and for this I used MicroProfile Config. MicroProfile Config is a small spec that standardizes the process of doing key-value-based configuration, allowing me to just say that I want a named value and let the runtime pick it up from the environment, system properties, or a config file as necessary. It wouldn't be difficult to write a configuration checker, but why bother reinventing the wheel when it's right there?

Same goes for having the SFTP server load at startup and be running consistently. If I did this on Domino, I could either put it in an NSF-based XPages app with an ApplicationListener and depend on the preload notes.ini parameter, or I could go my usual route and use an HttpService that manages the lifecycle. Neither route is difficult either, but they're both weirder and more fiddly than using servlet context listeners in a normal JEE app.

And then there's just the death by a thousand cuts: needing to use Tycho to build, having to deal with Eclipse Target Platforms, making sure anything using reflection is wrapped in an AccessController block to avoid Java policy issues, the nightmare of dependency management, the requirement to use either Designer or the okay-but-still-cumbersome Domino HTTP restart development cycle, and on and on. With a JEE app, all of those problems disappear into thin air.

The Main Hurdle

All that said, there's still the hurdle of actually implementing Notes native API access outside of Domino. At its core, what I'm doing is the same as what has been possible for years and years, initializing a Notes/Domino runtime in a secondary process. The setup for this varies platform by platform but generally involves either setting a couple environment variables for your process or (as is the case with the Domino Open Liberty Runtime) spawning the process from Domino itself.

Beyond setting up your process's environment, there's also the matter of initializing each thread of your app. On Domino, all threads are inherently NotesThreads, but outside of that there's some specific management to be done. You can call NotesThread.sinitThread() and NotesThread.stermThread() manually or run your code in specifically-spawned NotesThreads. I largely do the latter, making use of an ExecutorService to handle maintaining a thread pool for me. I then added some supporting code on top of that to let me run blocks of code as an arbitrary named user while retaining a cached set of sessions. That part wouldn't really be necessary if you're writing an app that doesn't need to act as different user names, but it's handy for something like this, where incoming connections must run as a user to enforce security fields properly.

Philosophical Advantages

Beyond my desire to avoid hassles and get to use modern Jakarta EE goodies, I think there are also some more philosophical advantages to writing applications this way, and specifically advantages that line up with some of HCL's stated long-term goals as well. Domino has long been a monolith, and that has largely served it well, but keeping everything from the DB all the way up through to the app-dev stack in the same bag means you're often constrained in your toolkits and deployment choices. By moving things just outside of the main Domino tower, you're freed up to use different languages and techniques that don't have to be integrated and maintained in the core. This could be a much larger jump than I'm doing here, and that's just what HCL has been pushing with the AppDev Pack and the associated Node.js domino-db module.

I think it's beneficial to picture Domino more as a dominant central core with ancillary servers and apps running just adjacent to it - not a full-on Microservices architecture, but just a little decentralized to keep areas of concern nice and separate. Done well, this setup is a lot more flexible and fault-tolerant, while still being fairly straightforward and performant. It's also a perfect match for a project like this that's geared towards implementing a new protocol - it doesn't even have to worry about HTTP SSO or reverse proxies yet.

So I think that this is where things are heading anyway, and it's just a nice cherry on top that it also happens to be a much (much, much) more pleasant way to write Java apps than OSGi plugins.

Another Side Project: NSF SFTP File Store

Nov 5, 2019, 4:12 PM

When I Know Some Guys kicked off, we bought a couple of Transporter devices to handle our file-syncing needs without having to rely on Dropbox or another hosted service. Unfortunately, Nexsan killed off Transporters a couple years back, and, though they still kind of work, it's been a back-burnered project for us to find a good replacement.

Ideally, we'd find something that would handle syncing data from our various locations transparently while also allowing for normal file access through some common protocol. Aside from the various hosted commercial services, there are various software packages you can run locally, like OwnCloud and NextCloud. I even got a Raspberry Pi with a USB hard drive to tinker with those, though I never got around to actually doing so.

Yesterday, though, I realized that we already have a fleet of privately-owned servers that replicate seamlessly in the form of our Domino domain. They also, conveniently, have nice capabilities for blob storage, shared user authentication, and fine-grained access control. What they didn't have, though, was any good form of file protocol. I'm pretty sure that Domino still has WebDAV built-in, but that's just for design elements. Years ago, Stephan Wissel created a project that works with file attachments, but that didn't cover all the bases I wanted and I didn't want to adopt the code base to extend it myself. There's also Karsten Lehmann's Mindoo FTP Server from around the same time, but that was non-SSH FTP and targeted at the local filesystem.

So that meant it was time for a new project!

The Plan

I initially looked at WebDAV, since it's commonly supported, but it's also very long in the tooth, and that has led to all of the projects implementing that being pretty old and cumbersome as well.

Then, I found the Apache Mina project, which implements a number of server protocols, SSH included, and is actively maintained. Looking into how its SFTP support works, I found that it's shockingly simple and well-designed. All of the filesystem access is based on the Java NIO packages added in Java 7, which is a pluggable system for making arbitrary filesystems.

Using SFTP and SCP means that it'll work with common tools like Transmit and - critically - rsync. That means that, even in the absence of an custom app like Dropbox, mobile access and syncing with a local filesystem come "for free".

The Project

So out of this was born a new project, NSF File Server. Thanks to how good Mina is, I was able to get a NIO filesystem implementation and SFTP+SCP server up and running in very little time:

Screen shot of the SFTP server in Transmit

In its current form, there aren't a lot of tricks: the files are stored as attachments to normal documents in a "filestore.nsf" database with two views, which allow for directory-contents and individual-file lookup while also being pretty self-explanatory to a Notes client. I have some ideas about other ways to structure this, but there's an advantage to having it be pretty basic:

Screen shot of the File Store NSF

Similarly simple are the authentication mechanisms, which allow for both password and public-key authentication based on the HTTPPassword and sshPublicKey fields in a person document, respectively (and maybe via LDAP in directory assistance? I never remember the mechanics of @NameLookup).

The App

Because this is a scratch-our-itch project and I'm personally tired of dealing with Domino's OSGi environment, the app itself is implemented as a WAR file, expected to be deployed to a modern Jakarta EE server like my precious Open Liberty. Conveniently, I have just the project for that as well, making deploying NSF-accessing WARs to Domino a bit more reasonable.

For now, the app is faceless: the only "web app" bits are some listeners that initialize the Notes environment and then spawn the SSH server. I plan to add at least a basic web UI, though.

Future Plans

My immediate plan is to kick the tires on this enough to get it to a point where it can serve as its original goal of a syncing SFTP server. I do have other potential ideas in mind for the future, though, if I feel so inspired. Most of my current logged issues are optional enhancements like POSIX attribute support, more-efficient handling with the C API, and better security handling.

It's also a good foundation for any number of other interfaces. A normal web UI is the natural next step, but it could easily provide, for example, S3 API compatibility.

For now, though I haven't gotten around to uploading a build to OpenNTF yet, feel free to poke around the code and let me know if any ideas strike your fancy.

Java With Domino After XPages

Mar 14, 2019, 3:10 PM

IBM and HCL held a webcast today to detail some plans for Notes/Domino V11. There were some interesting tidbits elaborating on things like the pub/sub support, and it'll be worth tracking down a recording of the event when it's available.

What's important for this series, though, is that this event served as the long-promised "roadmap" announcement for XPages. The roadmap is, in effect, option three: HCL plans to look into ways to reuse some existing XPages code, but in general you should be aiming to write your UIs in something else, either consuming REST services from an XPages container or accessing Domino data via another route (like the domino-db Node.js module and hypothetical Java gRPC client).

So we know the end of the path: not XPages. However, it's not like we're all just going to throw away our existing apps, so there's work to do determining how we're going to get there. The options remain pretty much what they were after CollabSphere last year, albeit now with the doubt removed. The first two options - returning to LotusScript or going to Node - have their advantages and disadvantages, and you could make a reasonable case for either. Personally, I'm not interested in going down those roads, though, and I think it's better for any app of reasonable complexity to dive into Java. Other members of the community and I have developed tools over the years to make it easier, and now's the time to take some of these steps if you haven't already.

Do Not Use Server JavaScript

Sever JavaScript was always something of a trap for app architecture. There's nothing inherently wrong with having a scripting language on your UI pages, and it certainly helped bridge some gaps, but the way it and Designer intertwined encouraged developers to create non-portable messes. If you're still writing SSJS, stop immediately.

Learn Proper Java

Java has been around for a long time, and the way to right "good" Java code has changed over time and varies greatly by your environment. Some aspects, though, apply generally, and it's useful to stay up-to-date on current practices. I don't know a better resource for this than Effective Java, which has been updated for Java 7-9 since I last read it.

Speaking of which, you should learn about Java 8 streams and lambdas - they're great. Julian Robichaux did a presentation on this topic back at Connect 2017, and the slide deck is very elucidating.

Adopt Standard Java Technologies

Last year, I created a project to bring some modern JEE technologies to XPages. These are some of the same technologies I've been talking about in my "XPages to Java EE" series and, while that project can't bring the full JEE development experience to XPages, using those tools will help you write code that, in some cases, could be directly dropped into a Java EE app with no modifications at all. There's a big asterisk when it comes to actually accessing Domino data, but that's a solvable problem as well (with some more development).

In particular, you should start writing JAX-RS services. Not only is JAX-RS an excellent and very-capable spec, but REST services are portable to absolutely any front end.

Adopt Automated Builds

Maven has been something of a bugaboo for XPages developers for a while, but doesn't have to be. Node development (server- or client-side) revolves around npm and various build plugins, and Maven is much the same thing. One of the biggest improvements I've made lately to all of my active XPages apps is to wrap the on-disk project for them inside a Maven artifact, using the NSF ODP Tooling. That project allows you to automatically build your NSFs alongside other parts of the project (such as OSGi plugins) without having Designer involved.

Check the example project in that repo, and stay tuned for a 2.0 release (probably) imminently.

Learn Other Toolkits

If you're just starting the process of figuring out what to do after XPages, it doesn't particularly matter which other toolkit you learn, as long as it's reasonably modern. If you take some time to learn how to make, say, a React app but end up going with something else down the line, the lessons you learn will apply very closely. A particularly-comfortable option could be to learn JSF, which has a common ancestry with XPages but has up-to-date capabilities.

Whatever it is, though, just learn some other toolkit.

Follow Channels and Accounts for Other Tech

Over the last couple months, I've started following a lot of Jakarta-related blogs and Twitter luminaries. This applies elsewhere - even if you're not using other toolkits yet, it's very helpful to start immersing yourself in the news and culture.

Don't Stay Still

The primary thing to take to heart is the importance of doing something. Unless you're planning to change careers or retire in the short term, you'll have to make one decision or another. XPages is not going to get meaningfully better, and even existing apps will get worse with time as browsers and technology change.

Other environments, though, are already leagues ahead and are constantly improving. Dive in; the water's fine!

Domino 10 for Developers

Oct 9, 2018, 1:17 PM

Tags: domino

So Domino 10 is upon us, marking the first time in a good while that Domino has had an honest-to-goodness version bump.

More than anything, I think V10 is about that sort of mark. Its primary role in the world is to state "Domino isn't dead" - not exactly coming from a position of strength for the platform, but it's the critical message that HCL has to sell if they're going to be viewed as anything but coroners.

Still, in addition to merely existing, V10 brings some changes that will help developers, particularly those - sadly - maintaining large legacy applications.


The addition that will have the largest immediate impact on developing codebases is, I think, DQL. I went into a bit of detail on this before and I think that that post is largely accurate, but the general gist of it is that DQL can be thought of as " but good", bringing practical arbitrary queries of non-FT data to Domino.

In its current form, it feels like a long-back-burnered passion project that's implemented in an effective way, bringing some of the benefits of arbitrary queries in SQL and new-era NoSQL databases without having to rewrite NIF or NSF storage.

UpdateAs Karsten Lehmann kindly pointed out, DQL is slated for addition to the LS/Java classes in 10.0.1 at an as-yet-unspecified time.

HTTP Methods in LotusScript

I just let out a heavy sigh after writing that header, but I get why they added these. A lot of Domino developers never left the desiccated-but-comforting embrace of LotusScript or are employed primarily to maintain Notes client apps, where using Java is possible but involves jumping over hurdles.

Network operations have been possible for a long time via OLE (on Windows) or LS2J, and I'm sure I'm not the only one who's had a "Network" LS script library sitting around for over a decade, but having baked-in methods is preferable. Moreover, neither of those mechanisms would work on iOS without a lot of additional work.

This is being billed as enabling all sorts of integrations, which I suppose is strictly true in that it's a bit easier to call HTTP methods in old code now. In practice, I think it will be mostly helpful for the little one-off situations where you have to call some web service to integrate with a product tracking app or the like.

iPad Notes Client

This definitely seems like another back-burner project that was brought to the fore in the HCL transition. They had iOS references in the Mac 64-bit C SDK years ago, and it only makes sense, since the existence of the Mac port at all meant the job was (sort of) half done. It's not out today, but it's logically tied to V10, and they've been expanding a beta over the summer.

Like the additions to LotusScript, it makes sense. I can't imagine that running existing Notes apps on an iPad will be a good experience, but it should be a cheap one, and it'll probably be good enough for at least some cases. They've intimated that there will be affordances in app design to improve the experience specifically for this client, though I don't envy the engineers who have to go in and implement those.

Node.js Support

Dubbed the "App Dev Pack", Node support will be coming in an Upgrade-Pack-like additional download, in the form of a Domino server addon to add a gRPC server combined with a domino-db Node module that I gather is designed to be familiar for Node+MongoDB stack users.

When this intention was first announced, I think that a lot of Domino developers figured it would be like XPages: a new design element or two added to the NSF, plus another runtime crammed into Domino's aging HTTP stack. The other big potential option was essentially a codification of the ExtLib DAS REST services into a wrapper package to be used in standalone Node apps.

The App Dev Pack is more the latter than the former, but the use of gRPC should make it more performant and flexible than just wrapping the existing HTTP services. I'll be curious to see how this shakes out in practice. XPages has been with us for a decade, but it still only captured a slice of the Domino development market, and it carried the advantage of being bundled right into the stack. Node is a very different beast, entirely unlike traditional Domino development, and I'm not sure how many existing Domino developers will make the transition. Ostensibly, one of the main benefits is to also attract new blood, which - well, maybe.

Having this is much better than not, and the notion of having a new RPC connection that doesn't have the local runtime requirements of NRPC is tantalizing.


Overall, this release definitely feels like a very pragmatic release. Just by virtue of its existence, it covers the base of "Domino isn't dead" in a way that's much better than the older mealy-mouthed messaging of "well, we don't have specific plans to cancel it". Additionally, though, the fact that most of the developer-facing improvements are for "old world" design elements is an acknowledgement that XPages didn't capture the Domino development world (and, probably, that HCL didn't hire the XPages team). The prospect of the community crawling back into the LotusScript cradle isn't great, but there's no avoiding the fact that there are a great many developers who never had a reason to do anything different. Not many cost-cutting IT departments let their developers re-learn their entire skillset when other departments are just asking for a new button on a form.

In an alternate universe, this would have made for a fine "Domino 9.5" release, but the wringer that the 9.0.1 era put us through demanded a full major version bump. I'll be curious to see how Domino 11 and so on shape up. If the "not dead" push works and it turns Domino's fortunes in the market around at all, it would give HCL room to turn it into a real platform again. That's a big "if", since it's a lot easier to get existing Domino developers excited than it is to get IT purchasers to sign the licensing checks, but time will tell.

DGQF and DQL as I Understand Them

Jul 26, 2018, 12:27 PM

Tags: domino dgqf

At CollabSphere this year, the big information coming from HCL was detail about the Domino General Query Facility (DGQF) and its associated language, Domino Query Language (DQL). They originally announced this a few weeks ago, but it was good to have had some time to let the dust settle and to see the specifics.

Because it was discussed alongside the domino-db Node.js package and because it's one of the first real new ways we'll interact with data in a Domino DB in a while, it's a bit difficult to identify just what it is and what it is not. Here's how I understood it:

What DGQF Is

DGQF is, at least conceptually, a "meta" layer on top of the existing NIF indexing facility. It doesn't provide a core change to the actual storage of documents, but instead treats existing view indexes as (roughly) analagous to both SQL table indexes and SQL views. It trawls through the design elements of a database to analyze their selection formulae and columns to use applicable ones as implicit indexes and also to allow access to arbtirary collections within queries.

Implicit Indexes

Other than the design collection and the "optimize document table" option in a DB, an NSF doesn't really have much in the way of indexing note contents by default. So, if you have a query asking for all documents where FirstName is Bob, a program has no choice but to look through every document for that key/value match. If, however, you create a view that has a column showing the FirstName field, you now have a much-faster index you can use. It's this sort of view that the DGQF picks up on implicitly, using them to accelerate queries: views showing all documents with either a default sort or "click to sort" column showing explicitly a field (and not a formula).

Access to Arbitrary Collection Data

For those qualifying views plus others, you can reference a view by name or alias to compare to a column value by programmatic name (often either the field name for simple columns or something like $4 by default for formulas).

"In" clauses

Additionally, you can use view (and folder, I think) names to refine queries for documents that are in one or more of these collections, equivalent to an "in" subquery or view reference in SQL

What DQL Is

In short, DQL is the human-readable query language used to access DGQF. It's reasonably SQL-like (though it is not SQL) and tends to look like FirstName='Bob' and in all ('Managers', 'Active Users'). This is the language you will use, and so "DGQF" and "DQL" will generally refer to the same thing in practice.

In practice, this is implemented as a new method on the Database class in each high-level language supported by Domino, plus a Node-styled variant in domino-db.

What DGQF and DQL Are Not

Since DGQF sits on top of NIF (and probably the FT index eventually), it's not a core change to data storage. Eventually, the same abilities and limits of Domino remain as they are with respect to this.

Additionally, DQL is, I believe, a query language only: it does not provide a mechanism for creating, modifying, or deleting existing documents. Instead, it is essentially a super-powered and much-smarter version of…): you can use it to find documents and the processing of them is up to your program.

That last point was a bit muddied by its pairing with the domino-db Node.js package: the Node.js package provides bulk operations that are paired with DQL queries, but that is a function of that library specifically, not DQL or DGQF.

Why It's Cool

Though it's not a reworking of the core NSF, what DGQF does do is abstract away a lot of the manual looping and lookups that we've always had to do, and it allows the system to optimize and do things more efficiently than when written out procedurally. So, while there's theoretically nothing that DGQF does that we couldn't do before, it allows us to do those things with far, far less code and with automatic optimization.

This brings Domino something that SQL servers have enjoyed for a long time. With a SQL statement, you can analyze the trouble spots of a slow-running query and add indexes to improve the speed, with the tooling helping to explain what's going on. DGQF+DQL brings this along for the ride: when you execute a DQL query, you have the option to dump out this "explain" output to see what specifically the facility did, which views it used, and how long each step took. So, if you have a long-running query, you can look to see if you can add an "index" view to automatically speed it up without having to change your code. And, since the language is an abstraction over the task of querying and not the sort of "burned in" process of a normal getNextDocument loop, it can be optimized and short-circuited by the underlying system without the developer having to know the decades of built-up knowledge of how to efficiently search a DB.

All in all, this is a very welcome addition to the server, and it certainly should improve a lot of common tasks.

A (Java-Centric) Domino Wish List

Jul 12, 2018, 12:04 PM

Tags: domino java

Seeing the information come out of this week's HCL "Golden Ticket" event has got me thinking about some of my wish-list items for Domino development, mostly in the form of enhancements for existing capabilities and entirely around Java (since that's what I do).

Quality of Life


For some reason, the lotus.domino classes ship without Javadoc or even variable-name information, leading to this trainwreck:

Designer has its built-in help, which is also on the web, but that's quite a few steps down. This is table stakes for a Java API and always has been.

Updated p2 Repository

Back in 2014, the XPages team uploaded a clean p2 repository of the XPages artifacts to OpenNTF, corresponding with the 9.0.1 release. This repository saves a ton of hassle when building Tycho-based projects or just setting up an Eclipse workspace. However, it's quite long the tooth, as there have been several Notes.jar additions not included in there, and, in FP10, a significant upgrade to the undergirding OSGi framework.

I ended up writing a script to generated an updated version, but I don't have the legal ability to publish the results anywhere for easy consumption, meaning it has to be done manually and configured for each build environment. It would be a great convenience if there was an official package (ideally including the Designer plugins as well) and, even better, hosted on OpenNTF so that we could reference it by URL as we do for Eclipse releases (and require users to accept a license first).

Mavenized Repository

The p2 repository is good for Tycho-based projects, but, especially when targetting Domino is only one part of a project, it can be much more convenient to use "normal" Maven projects with maven-bundle-plugin. However, those projects can't use p2 repositories as such. For Darwino's needs, I ended up writing a tool in the (available-for-free) Darwino Studio plugins to Maven-ize a Domino p2 repository, but that hits the same snag as above of requiring manual setup in each instance.

This is another case where my preference would be on the OpenNTF Maven repository (plus Javadoc Jars, naturally).

Extension Library Source

The latest Extension Library release on OpenNTF is from FP9, while the latest on GitHub is from the FP7 era. FP10 shipped with a newer version of indeterminate nature. It'd be good to have this on both of those sites and, like with Javadoc, have source bundles shipped with the product in a way that is picked up automatically by Designer and Eclipse.

Source Bundles for Third-Party Components

The source for the undergirding Equinox stack is available, but it would be best to have, as an adjunct to the updated p2 repositories, the source bundles for the actual versions used so that we don't have to cobble together a platform from Eclipse's repositories.

Open-Source the Rest of the Stack

Having XPages, the Expeditor husk, and the other miscellaneous doodads that make up the proprietary layer as open source with an Apache-compatible license would cover a lot of the above and also be of tremendous use for XPages and non-XPages apps alike that run on or with Domino. I have a hard time imagining that it would lead to a lot of community-driven improvement, but it may do some (I'd have a few words to share with the file-download control, for example), and even just as a static release would be a significant boon.

Domino Connectivity in Eclipse

An idea I've been toying with lately is to make an Eclipse plugin that allows you to add Domino servers to the "Servers" view and control them to some extent. The basics would be to start/stop/restart HTTP, but the stretch goals would be to open a console view, get a list of running modules, integrate with the existing "load bundles from PDE" support, and, ideally, an outright "Run on Server" command for OSGi bundles and NSFs. However, I have so much on my plate that I'm not sure that I'll get to this any time soon unless I get a real itch some weekend.


Refreshed JVMs

Feature Pack 8 brought Java 8, a vital step forward. However, since then, Oracle moved to a faster release cycle for Java and the JRE is now at version 10. Domino uses IBM's JVM variant, J9, which they recently moved to the Eclipse foundation as OpenJ9, where it has... sort of been keeping up, I think?

In any event, this increased pace of change has meant that the Java 8 honeymoon is over, and Domino development again requires special consideration when using current tools. I have no idea how complex the integration between Domino's tasks and the underlying JVM is, but my ideal would be to have constant or near-constant parity.

Servlet API 4.x

After the JRE version, the most important foundational element of a Java web app is the servlet API release. The current version is 4.0.1, while Domino supports 2.5 (or 2.4, maybe?). The good news is that the Java/Jakarta EE world seems to be used to lagging versions here, and 2.5 is a minimum version for a lot of current tech in much the same way that Java 6 was until somewhat recently, but there has been quite a bit added in recent years.

Presumably, a reason for the lag is the implied requirements of newer versions, such as WebSockets and HTTP/2 support, that would require heavy modifications to the core Domino HTTP code. Honestly, the more practical route is almost definitely to just use a different JEE server paired with CrossWorlds, some Java wrapper for the GRPC stuff HCL has been talking about, or (best of all) a Darwino app replicating with Domino, but still. WebSphere Liberty is actually really nice, by the way.

Refreshed Equinox

Like with the underlying JVM, the Feature Pack 10 update to a Neon-based OSGi/Equinox framework was a critical shot in the arm for the platform, but it is now also two major versions behind. This is a little less critical, since Equinox brings a bit less to the table for our needs and Neon is "new enough" for now, particularly on the server side, but it'd still be proper to keep pace.

Odds and Ends

Non-OSGi JEE Support

The Equinox framework that Domino uses is quite capable, but there's no pretending that OSGi-targetted development has its share of headaches. Most Java apps just target plain-old .war files and don't impose any particular requirements on the build process. Java development is a much more pleasant experience when you can just toss in any Maven dependency and not have to think about building a target platform for Eclipse or jumping through bundle-resolution hoops. I really like OSGi in theory, but I can't pretend that non-OSGi development isn't much smoother.

Domino technically supports "regular" servlets currently, but, uh, here's a snippet from the current documentation on that:


Full Extension Manager Support for Java

JAVADDIN/DOTS added a lot of EM hooks, but it doesn't cover the full suite of capabilities that a C addin can provide, such as authentication handling. Having this be fully accessible from Java would be useful even when treating Domino just as a data store and not as an app server.



I'm sure I could come up with more, but that's probably good for now. All easy, right?

First Steps to Code Coverage Analysis in Domino Plugins

Nov 9, 2017, 8:53 AM

Tags: maven domino java

I'm always interested in getting the computer to tell me how to tell it what to do more successfully, and, to further that pursuit, I've started taking an interest in code coverage.

If you're not familiar with the term, "code coverage" refers to reporting on which lines of code were actually executed during runtime, most commonly in association with unit tests. Eclipse (and presumably other IDEs) has support for this, and I've decided to give it a shot.

Since I'm starting this out in the context of Domino plugins, there are more wrinkles than in most tutorials. Namely, the test suites I've written run exclusively through Maven instead of the Eclipse UI due to all the Notes environment setup, so I can't just use the normal UI tools to gather the data. Fortunately, Eclipse's EclEmma will work just fine with the output from a Maven project, as long as you configure it properly. I looked around for a while to find the right combination of tools to use, but it ended up being fairly simple to configure basic output that can be consumed in Eclipse's Coverage view.

There are two main additions. First, add the jacoco-maven-plugin to your root project's block:


In normal cases, that would suffice. However, since the test configuration I have for Notes overrides the argLine property of the Tycho test runner, there's another step - add the tycho.testArgLine property manually into those blocks, such as in the Windows profile:


					<argLine>${tycho.testArgLine} -Dfile.encoding=UTF-8 -Djava.library.path="${notes-program}"</argLine>

Once that's configured, running the test suite via Maven will create a new file in the target folder of the test plugin: jacoco.exec. This file can then be consumed in Eclipse by opening the "Coverage" view:

Eclipse's Show View window

In that view, right click and choose "Import Session..." and point to the data file. Click "Next" and check the projects+source folders from your workspace you're interested in analyzing. When you click "Finish", it'll do two things. First, it'll fill the Coverage view with statistics from your run:

Code Coverage stats

(We have a lot of work to do fleshing out our test suites for this one)

Secondly, it'll start highlighting your code to show you what code is executed, which branches are only partially covered, and which lines are skipped entirely. For example (ignore the sickly color scheme - I need to work on that):

Code Coverage example

This shows how several of the if branches are only tested in one direction, while the "Faces" block is skipped entirely. That also shows some of the trouble with testing XPages-run code: the Tycho environment can't reproduce the XPages environment fully, so some branches aren't testable in that way. I haven't looked into the possibility of gathering similar data from JUnit for XPages, so perhaps that's possible.

For now, though, this will have to do. And, like with these other "code improvement" techniques I've integrated lately, there's a lot of potential tedium - juggling when to write a test to cover some code that will obviously always work just to improve the highlighting vs. just focusing on the low-hanging fruit - but I expect that it will be a nice addition to my workflow over time.

Change Is In The Air

Aug 26, 2016, 5:41 PM

Tags: xpages domino

During last week’s MWLUG, there was a clear sense that things are a little different this year. Dave Navarre dubbed the technical implications “platform agnosticism”, while I geared my presentation towards the feeling that change is in the air.

This is not totally new. Red Pill Now cast aside the XPages UI layer and most of the assumptions of Domino development to move to a new level; PSC's presentations have long developed a polyglot tone, and this ramped up this year; and people like Paul Withers have been growing with tools like Vaadin.

It's not too important to dive into the specifics of market forces or the shifting sands of technology, and the platform defensiveness that we tend to wear as a cozy blanket doesn't serve anyone properly. Our beloved Domino app-dev platform has grown pretty long in the tooth and it doesn’t seem like it’s in for a revitalization.

The situation is, fortunately, an opportunity.

One of the things I hoped to convey in my presentation is that, though the prospect of learning some of the ever-changing array of modern development tools is daunting, it is also exhilarating and profitable both professionally and personally. As insane as the tangled web of server frameworks, JS optimizers, language transpilers, automation tools, dependency managers, and so forth may seem, particularly compared to the simple days of Notes client development, there is a great deal of good news.

The popular tools are awash in documentation, with clear examples for doing basically whatever you will want to do. There's also a lot of conceptual overlap and familiarity: if your tool of choice loses favor, it won't be a start from square one to learn the next. And it's not required to learn every single thing that comes along. If you build yourself a Java web stack using, say, Spring that does the job, it's not also necessary to learn every single new client-side JS app framework that comes along.

And, in the mean time, there's a lot of great work left to do with Domino. There are XPages applications in use and development, and these will go a very long way. Domino remains a very capable platform, and the path through XPages can be a very natural lead-in to other technologies, especially if you focus on the aspects that carry on: Java, data separation, REST services, and so forth.

For my part, I, too, still have great work to do on Domino and XPages, but I'm also expanding beyond it with eyes open. As I mention frequently, I believe that Darwino is the best path for a number of reasons. When I have the opportunity, I plan to start getting into the meat-and-potatoes reasons why and examples of how to actually use the thing. Time permitting, I hope to have a series at some point for converting my blog over to a Darwino+JEE application, and I'll share my process of picking my tools and replicating with its current NSF form as I go. If all goes well, it should serve as an example of taking an existing XPages app and transforming it into something new.

This is an opportunity, and it's an exciting time.

Postscript: This is the optimistic take, granted, and some people’s situations are a bit more dire. Admins, I imagine, are in a strange spot (I hope you’ve been brushing up on ancillary tools!), and a lot of companies are doing a lot of hand-wringing about the future for app dev and maintenance as well, particularly those with a heavy Notes-client dependency. My point is that it’s not necessary to get too mired in the doom and gloom.

Change Bitterness and Accidents of History

Jun 17, 2016, 3:16 PM

Tags: domino

It's pretty easy to see that change is in the air for Domino types. It's been taking a number of forms for a while now - the long delay since the release of 9.0.1 and associated aging of the tools and infrastructure have led to a series of forced adaptations for developers and administrators. Developers, for example, have had to keep light on their feet to adapt to new browsers and devices that the framework doesn't automatically support, as well as a shift toward manually including jQuery and other tools that have a bit more wind at their back than Dojo. Administrators, for their part, have had a series of heart attacks related to SSL and other security matters, usually involving a a lot of noise followed by (unfortunately, I feel) a good-enough patch from IBM.

That sort of thing isn't likely to get any smoother. In large part, that's entirely distinct from anything IBM does: the genie's out of the bottle when it comes to fast-moving platforms, and the best we can hope for is a sort of still-moving linga franca that can be sort-of-stable on the majority of targets. But then part of that is Domino: for all its virtues, it hasn't adapted for the modern world, no matter how much some of us would have liked it to. And that's had some negative side effects on us as a customer base, side effects that manifest as a gut-reaction rejection of the modern ways of doing things.

Take the SSL thing, one of my soap boxes: though the immediate problem could be summarized as "IBM should update their SSL stack", the larger issue it exposed was that our beloved monolith is mortally vulnerable to a single component falling behind. And, in fact, it made clear that we're spoiled by the approach: many of the reactions were basically that we shouldn't have to worry about things like reverse proxies or the general notion of distinct systems for web front-ends and the app server. And that initial rejection of the hassle has implications, limiting the average Domino installation's capacity to scale for load balancing or failover in a smooth way, things that come almost "for free" with a reverse-proxied setup common among almost all other app servers.

Developers have it worse: the "power user turned developer" history of Notes and the particular peculiarities of the platform* have left us desperately behind the baseline for modern development. The tooling has coddled us into preferring inline scripts and procedural programming to structured code organization, into viewing a thoroughly-staid language like Java as something outrageously complex, of only begrudgingly adopting SCM due to the platform-induced hassle, and of almost entirely ignoring automated testing. And similarly to the SSL discussion, the historic "one tool for every job" nature of Domino leads to natural pushback when faced with other platforms.

And I get why! And I feel it too. It's a real PITA to now always have to be on the lookout for some new point release of iOS or Android to break drop-down boxes or something, as opposed to years of deploying Notes apps that looked and worked identially across every version forever (pretty much). It's also a drag to run into situations where the problem is "easy" on Domino but more cumbersome elsewhere, like platforms that farm out their FT indexes to distinct servers, or don't include document/record-based security. That makes it very easy to become blinded to the tradeoffs, though. It may be nice that Domino is a one-stop-shop for so much, but it's a shop that requires Designer, that makes it very difficult to use third-party-dependency systems like Maven (even within Maven projects), that lags in DB features found elsewhere, that is only awkwardly accessible from other app-dev frameworks, that has an API that's a bit older than Windows Me, and that essentially never showed up for the modern development conversation.

Domino has always had a lot to recommend it, and XPages has carried us very far. And hey, this is enterprise software - even if there's never a major new version, there'll be paying work forever. It just may not be the kind of work you want to do, and it is almost definitely not truly healthy for the companies paying for it. So what I recommend is that you have a plan. The good news is that there are a great many next steps that build smoothly on existing Domino knowledge and, potentially, infrastructure. Certainly, I'm thoroughly biased in the direction of Darwino, but that's one of many. You could also do worse than learning a mature platform like Ruby on Rails (heck, you could run that on JRuby on Tomcat or WebSphere). Take some time to learn about reverse proxies and modern web-server setups. Basically: something. Just do it with an open mind, and don't balk at the first thing that's more complicated than the Domino equivalent. I think a bit of that will serve you very well.

* And, to be fair, of the overly-conservative nature of enterprise programming.

Other than, of course, being one of the progenitor NoSQL databases.

Platform Defensiveness

Jul 18, 2014, 8:50 PM

Tags: domino

If you were a Mac user in the 90s and early 2000s, life could be tough. Though you loved your platform of choice and could see its advantages plain as day, you were in a distinct minority. The world was full of people ready to line up to explain why you're an idiot on a sinking ship. And the part that stung was that they weren't always wrong: for every person talking out of their ass about how great having a menu bar in every window was or the joys of running ancient line-of-business DOS apps, someone else pointed out the very real stability and performance issues that plagued classic Mac OS and OS X respectively, the fifth-class-citizen status among game developers, or the very-real possibility that Apple would join Netscape and Be under the treads of Microsoft's then-indomitable war machine.

The recourse for users was reflexive defensiveness. Mac market share is down to 2%? Well, that's the good 2%. Windows 2000 is actually stable and fast now? Well, the UI is still second-rate. Schools are switching en masse to PCs? Boy, they'll regret that when their cheap Dells give out! Much like the slings and arrows thrown at the Mac community, these defenses contained just enough truth to soothe the wounds, but it didn't matter. Maybe the community circling the wagons staunched the bleeding a bit, but the thing that really mattered was Apple breaking its own malaise and making great products again.

Unsurprisingly, what has me thinking about this is Domino. Specifically, the popularity of a post on Chris Miller's blog about Denny's apparently switching away from Notes. Now, I don't have a gripe with the post or the specifics of the situation - for all I know, Denny's is making the worst decision of its corporate-IT life. What bothered me is the paint-by-numbers way the story echoes through the Domino community. We've seen these things go around before, and the components are familiar when Company X decides to ditch Notes/Domino for Competitor Y:

  • "Just wait until they see the REAL cost of Y! Then they'll be sorry!"
  • "They must not be thinking about all their crucial apps! Y doesn't do that!"
  • "Their stated reasons are wrong! They said Domino doesn't do Feature Z, but Domino is actually the BEST at Feature Z!" (nowadays, replace with "Use some other IBM product to get Feature Z!")
  • "This is the work of some clueless IT manager who's just following the trends!"
  • "They must be using an ancient version of Notes! Modern Notes clients are exemplars of memory efficiency and clean design!" (alternative for server admins: "They must not have configured their mail environment properly!")
  • "I betcha they'll still be using Notes for their crucial apps in ten years, 'migration' or no!"

Much like the defenses of classic Mac OS, these all contain kernels of truth, and maybe Company X would indeed be better off sticking with Domino. But are these really claims you want to hang your hat on? The last one in particular is telling. Does it really fill your heart with pride that companies are going to be saddled with ancient, un-migratable code until the sun goes nova? There's a reason why "modernization" is such a big topic for Domino developers: the unfathomable mass of legacy Notes client apps is a severe issue. Whether you attempt to deal with it via clever en-masse approaches, by using remote-desktop workarounds, or by dragging apps one-by-one into the present, there's no getting around the fact that depending on Notes client apps is now an undesirable condition for a company.

So what's to be done about it? Well, for the most part, that's IBM's job. But for an individual developer, it's better to focus on why you should build a Domino app today, not decades ago. It builds on the past, sure - an XPages app I've been building for a client brings together data from dozens of their existing apps built over years in ways that would be much more difficult on other platforms. But even better is to acknowledge reality with clear eyes. Google Apps are really good, and they work on everything! Exchange provides a better mail/contacts/calendar experience for varied clients than Domino does. Non-XPages web-dev environments are brimming with surprising features and deployment has gotten good in recent years. The advantages of other platforms are not necessarily enough to be worth a switch, but they still exist.

Personally, yes, I care whether Domino ends up flourishing, but I profit more from addressing reality as it is rather than immediately dismissing bad news as illegitimate.

The Trouble With Developing on Domino

Jul 8, 2014, 10:57 AM

Tags: domino

The core trouble in my eyes with developing on Domino is that it is unloved for what it is. Not so much by customers (I have no interest in whether companies use Exchange or SharePoint or whatever), but more by IBM. The situation reminds me of Apple with products like WebObjects, Xserve, and Aperture: there's clearly a dedicated and talented team behind it, but the organization's heart isn't in it. The analogy is inexact: first, Apple's overarching motivations and strategies are much easier to grok than IBM's; second, enterprise software never really dies - it just goes all COBOL and lingers in maintenance somewhere forever (did you know OS/2 is still a thing?).

So Domino is still around, and still exists, but is generally positioned as something for large companies to use for mail, or continue using if they happened to start doing so decades ago, or as an adjunct to Connections. But it's like with Lotusphere's rebranding: when (dev conference) + (biz conference) = (biz conference), the algebra to figure out the perceived value of (dev conference) isn't difficult. To a certain extent, this is just how IBM does business: they talk to large organizations' higher-ups and make their sales that way, not by being outwardly compelling. However, on this front, Bluemix's entry has provided a telling counter-example: though there's still the stock over-puffed business page, the site talks directly to developers using a layout and writing style that an actual human being might enjoy. They even have honest-to-goodness Developer Evangelists!

It leaves us, as Domino developers, in an awkward position. We have a full-fledged NoSQL server with an integrated Java dev stack, albeit one without a guiding soul. We have an easy-to-install, flawlessly-clustering, cross-platform server that is perpetually 20% away from being perfect for the modern world. Large portions can be charitably characterized as having been in maintenance mode for a long time: the core API, SSL in all protocols, calendar/contacts connectivity, indexing, the admin client, and so forth. The bad news is easy to perceive: less vendor interest and customers driven primarily by inertia make for a poor career path. But the point of this post isn't doom and gloom! The way I figure it, there are a number of bright sides to explain why I and others continue to develop on this platform:

  • There's always a chance that Domino will be less "Xserve" and more "Mac Pro", a product seemingly at death's door for years until it was given a fresh breath of life.
  • As Paul Graham pointed out years ago, when you're writing server-run software, you can use whichever platform you'd like. Though, all else being equal, it's nicer to have a popular platform, if you can use the tool to do greater work with your time than you would elsewhere, it's worthwhile.
  • Due to Domino's nature, almost all of its (technical) problems are solvable even if IBM never touches it again. Some of them, like the API, are things where the community has already built a workaround. Some, like SSL for HTTP, have seen IBM package their own workaround with the server. And the rest, like NIF and CalDAV/CardDAV support, linger as perfect "if I had time" projects.

The last one is crucial for our position: though a totally unsupported problem would eventually fall prey to incompatibility with newer operating systems and machines, Domino has reached a point in the last couple years (thanks to the extensibility API in 8.5.2 and the vital bug-fixes in 8.5.3 and 9) where it's an agglomeration of replaceable parts. There are enough hooks and APIs to - with varying degrees of difficulty - take advantage of the core strengths of the platform while dramatically improving the components on top. That's not enough to last forever, but it doesn't have to be. Apps built in the mean time will still run and improve, and any time spent programming on the platform is valuable experience that can be applied elsewhere, whether directly or more generally.

Just don't ask about licensing.

Another Round of URL Fiddling

Jun 19, 2014, 5:59 PM

Tags: domino urls

I was working on another installment of "Be a Better Programmer", but the result isn't where I want it to be yet, so it'll have to wait. Instead, I'll share a bit about my latest futzing around when it comes to improving the URLs used in an XPages app. This has been a long-running saga, with the latest installment seeing me stumbling across a capability that had been right under my nose: the ViewHandler's ability to manipulate the URLs generated for a page. That method worked, but had the limitation that it didn't affect navigation between pages.

As it turns out, there's a slightly-better option that covers everything I've tried other than navigation rules specified in faces-config.xml (but who uses those in XPages anyway?): the RequestCustomizerFactory. This is one of the various classes buried in the Extensibility API and present in the XSP Starter Kit, but otherwise little-used. In the Starter Kit stub (and in a similar commented-out stub in the ExtLib), it can be used to add resources to a page without needing to specify them in code - say, for a plugin to add JS or CSS libraries without being specified in a theme.

However, there are other methods in the RequestParameters object that would appear to let you shim in all sorts of settings on a per-request basis - different Dojo implementations, different theme IDs, whatever "debugAgent" is, and so forth. The one that interests me at the moment is the UrlProcessor. This is aptly named: when a URL is generated by the XPage, it is passed through this processor before being sent to the browser. This appears to cover everything other than the aforementioned stock-JSF-style navigation rules: link value properties, form actions, normal navigation rules, view panel generated links, etc..

Following my previous desire, I want to make it so that all links inside my task-tracking app Milo use the "/m/*" substitution rule I set up for the server. The method for doing that via a UrlCustomizer is pretty similar to the ViewHandler:

package config;

import java.util.regex.Pattern;

import javax.faces.context.FacesContext;


public class ConfigRequestCustomizerFactory extends RequestCustomizerFactory {

	public void initializeParameters(final FacesContext facesContext, final RequestParameters parameters) {

	public static class ConfigUrlProcessor implements RequestParameters.UrlProcessor {
		public static ConfigUrlProcessor instance = new ConfigUrlProcessor();

		public String processActionUrl(final String url) {
			String contextPath = FacesContext.getCurrentInstance().getExternalContext().getRequestContextPath();
			String pathPattern = "^" + Pattern.quote(contextPath);
			return url.replaceAll(pathPattern, "/m");

		public String processGlobalUrl(final String url) {
			return url;

		public String processResourceUrl(final String url) {
			String contextPath = FacesContext.getCurrentInstance().getExternalContext().getRequestContextPath();
			String pathPattern = "^" + Pattern.quote(contextPath);
			return url.replaceAll(pathPattern, "/m");


The method of adding this customizer to your app directly is, however, different: instead of being in the faces-config file, this is added as a "service", as mentioned in this handy list. For an in-NSF service, you accomplish this by adding a folder called "META-INF/services" to your app's classpath (you can add it in Code/Java) and a file in there named "":

In this file, you type the full name of your class, such as "config.ConfigRequestCustomizerFactory" from my example. You'll likely have to Clean your project after doing this to have it take effect (and possibly in between each change to the code).

Now, my current use is very limited; the UrlProcessor has free reign for its transformations and is passed every URL generated by the XPage (except hard-coded HTML, unsurprisingly) - you could change URLs like "/post.xsp?documentId=foo" to "/blog/posts/foo" without having to code that explicitly in your XPage. You could write a customizer that looks up all Substitution rules for the current Web Site from names.nsf and automatically transforms URLs to match. You could go further than that, trapping external links to process in some way in the "processGlobalUrl" method. The possibilities with the other methods of the customizer go much further (and are largely undocumented, to make them more fun), but for now just fiddling with the URLs is a nice boon.

Pretty URLs: A Step in the Right Direction

May 9, 2014, 8:20 PM

Tags: domino

A long time ago, I mused a bit about the URL problem in XPages. The core problem then, as it is today, was that XPage URLs are ugly as sin. Domino URLs were never pretty, but XPages took them back a few steps. Just look at this normal-case monstrosity:

Barely any part of that has anything to do with the task at hand! Some parts - like the folder path and NSF name - are only tangentially related, while others - like "action=openDocument" are little more than an implementation detail. The inhuman UNID gets a half-pass by virtue of its value as a cluster-friendly identifier, but it would be better as a human-readable unique key.

What you'd REALLY want would be a WordPress-and-others-style URL like this:

Now THAT'S a functional URL: every part of it is both explicable to humans and useful to computers as a unique identifier. More importantly, it establishes a clear hierarchy: any part of it can be traversed to get a conceptually-useful URL (assuming your app implements it). "/blog/2014/5" has a clear meaning: "give me all entries in the 'blog' app for May 2014".

Domino provides little assistance in getting to this point. There are web rules, yes, but that leaves two crucial problems: 1) your app needs to know about these rules, so it generates nice URLs and not eldrtich horrors and 2) you need to set up a server-level config for every web site + function combination. You can't just say "send all requests for /blog to so-and-so app" and then have that app handle everything past that. This is something of an impassable brick wall: Domino's original URL router is still in full effect even in the most modern of apps, and so the XPages side of an app doesn't even hear about a URL request unless it's in the form of "app.nsf/somepage.xsp" or "app.nsf/xsp/whatever" (and the latter form is something of a Wild West full of resource providers, servlets, and who-knows-what else).

But there's a twinkle of hope: because you specify all URLs in an XPage as relative to the app, that means that there's a post-processing service in there that translates a URL like "/foo.xsp" into a full server-relative URL like "/somedir/app.nsf/foo.xsp". That service has a name: ViewHandler. ViewHandlers have been one of my preferred tools in XPages ever since I discovered their utility in instantiating my soon-to-be-renamed "controller" classes. In addition to their role in creating pages and potentially intercepting page requests, ViewHandlers serve a crucial role: providing resource and action URLs. In XPages/JSF parlance, an "action" URL is what shows up in the form tag on the HTML result, while a "resource" URL is, well, basically everything else. When a control on an XPage specifies a URL, it passes it through the resource method to get the "real" URL. Normally, these are turned into the normal ".nsf" paths, but there's no reason they have to be. Here's an example of the two methods in my project-tracking app (named "Milo"):

public String getResourceURL(final FacesContext context, final String resource) {
	if(!isGlobalResource(context, resource)) {
		// Then switch it to "/m/whatever"
		return "/m" + resource;
	return super.getResourceURL(context, resource);
public String getActionURL(final FacesContext context, final String pageName) {
	return "/m" + pageName + ".xsp";

I set up a substitution web rule on the server to translate "/m/*" to the full path of the DB. Because of this change to the view handler, now every <xp:link/>, <xp:image/>, theme resource, etc. uses the "pretty" path. Assuming I don't run into any major down sides, this is a huge piece in the puzzle! Now I can write my apps normally - referencing XPages, resources, etc. using the same portable, app-relative syntax as usual - and then let my ViewHandler determine if it should clean up the URL. This is a nice step above my second solution, which required either a special EL pattern or a custom control to translate the URLs; now that job can be passed up a layer of abstraction.

There are still two problems I know of standing in between me and using this everywhere: automatic detection of server support (e.g. reading names.nsf for matching web rules) and navigation between pages. Though the ViewHandler covers normal links and form action URLs, it doesn't handle navigation between pages based on navigation rules. I don't expect that to be an eternal problem, though; JSF has hooks for this sort of thing, so it should be a matter of figuring out how to use it there.

Overall, it's still something of an ugly solution: where other platforms have clean routing configs, XPages has a hodgepodge of server-level settings and in-app shims. However, I hope to turn it into a worthwhile combination of automatic configuration (by reading names.nsf) and clean declarative settings that I can build into the Scaffolding project.

Domino the Identity Server

Feb 11, 2014, 9:10 AM

Tags: domino

As seems to happen a lot lately, my fancy was struck earlier by a Twitter conversation, this time about the use of Domino as a personal mail server. Not only do I think there's potential there, but it should go further and be a drop-in replacement for personal mail, calendar, and contacts storage.

I think there's tremendous value in controlling your own domain and the services on it, without being permanently attached to someone else's name for an email address (your school, your company, your ISP, Google). This is good not only for personal freedom, since it lets you pack up and move at will, but also for security, since a large third-party mail service is a particularly juicy target.

Unlike Domino's inherited-but-abandoned place as the preeminent NoSQL server and replicating app-dev platform, Domino is just barely shy of being this server. You could already hit the three main services reasonably well by using the Notes client or iNotes, but actual humans shouldn't have to do that. It's already (more or less) there for mail with IMAP, while support for CardDAV for contacts and CalDAV+public iCalendar feeds would round it out for the other two pillars. Technically, the only things standing in between the existing open-source WebDAV plugin for Domino and this imagined future are the complexities of plugin development and the RFC.

The other main aspects that could make this great are further refinements to Domino's existing capabilities: a bundled spam filter (say, one of the open-source tools that can do the job already) and a strong configuration focus on creating an SSL-secured public-facing server. Non-SSL variants of IMAP, POP (if you must - that could be removed entirely), and HTTP should be off by default and the configuration should encourage you to acquire SSL certificates with your own private keys (the Server Certificate Admin would need a revamp for proper key size and ciphers), as well as S/MIME certificates to tie with each user for signed/encrypted mail outside the Notes client. Though Domino's history with the NSA is... checkered, it's still remarkably well-positioned to provide a secure foundation to deter snooping eyes. Certainly, running a Domino server own your own or a rented/virtual server is leagues better than a fully-managed service in this respect.

Having those features in place and smoothly integrated with a nice setup assistant would make for a very compelling product: an all-in-one, easy-to-install server that runs on several modern OSes and handles secure replication across physical locations at already-actually-affordable prices. Admittedly, I don't know how compelling that product would be for IBM's accountants, but it's certainly compelling for me, and the world could use more decentralization like this.

How I Want To Use Domino, Take 2

Oct 30, 2013, 4:10 PM

  1. How I Want To Use Domino
  2. How I Want To Use Domino, Take 2

A while back, I wrote a post about how I wanted to use Domino. The gist of that was that I was enamored with the idea of using Domino as a back-end database, but not necessarily as a app-dev platform on its own - basically, how you would use a SQL or NoSQL database. Since then, I've doubled down on my use of XPages as an app-dev platform with many advantages, but I still find it very useful to imagine Domino not as "Notes apps on the web, now with a modern coat of paint", but as a collection of related but not mutually-required components competing with other web-dev stacks.

I started working on an updated take on that post, inspired by some recent posts and chat conversations I've had (and I may return to it), but then I ran across Grand Decentral Station, which is a vision of an OS/app-dev platform taking the best of the lessons of the last decade and turning them into a coherent platform. It's a compelling vision, and reading it made me realize something: most of those goals are describing Domino, or, more accurately, the Domino that could be. Take a look down the list and see how many points Domino goes about 80% towards:

  • App Installer & Updater: though Domino doesn't really handle app versions, the deployment strategy is nonetheless quite good, with all app code contained in a distinct NSF, not a bunch of files strewn in a couple directories.
  • Sandboxed apps: again, Domino doesn't quite sandbox apps, but appropriate use of ACLs can bring you close.
  • Mail server: I hear tell that Domino is capable of acting as a mail server.
  • Calendar server: with proper CalDAV support, Domino could act as a real calendar server for non-Notes clients like OS X and iOS.
  • Addressbook server: like with the calendar server, this is just a matter of adding CardDAV support.
  • Asset handling: CSS and JS optimization in XPages is a huge step in this direction.
  • Avatar server: the Directory can already act as a profile-picture server for Sametime, and something like mypic could standardize this use.
  • vCard server: well, it already serves LDAP.
  • Unified sessions: done.

And to cap this off, the prescribed per-app database is CouchDB, which is already modeled on Domino. And, of course, it already has a standard API for email, which conveniently doubles as a method of cross-app messaging in some cases, and its replication and clustering are top-notch. It's not, itself, an OS, but its fairly-cross-platform nature means that that problem is already "solved": install Domino on the server platform you're most comfortable with and it acts basically the same.

Of course, that final "20%" is, as always, the crux of the problem. Domino is only really a fully-integrated mail/contacts/calendar server when you use Notes or iNotes, XPages and legacy Domino dev are really the only games in town if you want to maintain the benefits of the NSF package, agents aren't integrated with the XSP environment and DOTS isn't a real replacement yet, there are still a number of items on the list not at all touched on by the existing stack, and licensing basically removes Domino from consideration for app development for anyone not already mentally invested in it. But hey, one can dream, no?

Domino Wishlist, September 2013

Sep 20, 2013, 1:06 PM

I joined in a tweet from Paul Hannan earlier about desired Domino features (that's a very worthwhile thread to read - lots of great ideas from others) and it's had me thinking about more on my list. Fortunately, I have a blog for this kind of thing! I'm leaving off a couple big-ticket items like Eclipse-4.x-based Designer on the Mac because IBM may not be required for that.

  1. Make $WSIS not trigger crummy SSL behavior (really, first-class support for proxies in front of Domino, not just IHS/Windows)
  2. Direct access to note item data via Java
  3. Newer JRE
  4. Direct creation/update of NSFs from git server-side
  5. New XSP model-object framework
  6. New views, though that may be our job
  7. Aggressive, small-company-friendly licensing to compete with OSS stacks
  8. NSF-based SSL keychains
  9. NSF-based jvm/lib/ext distribution
  10. "Sensible defaults" mode for clean-install servers to get appropriate memory/cache/performance settings on modern setups without learning dozens of secret notes.ini settings
  11. apt-get-based distribution and updates for Domino on Debian/Ubuntu
  12. Improved XPages file-upload and -download controls to handle modern file uploading and provide better hooks for non-dominoDocument bindings
  13. Bootstrap theme for XPages. Done - thanks!
  14. Built-in default support for authentication fields in LDAP
  15. init.d startup script installed with Domino, because seriously, who ships server software without one?
  16. OS X build of Domino 64-bit
  17. Built-in Firefox sync server, because that would be a nice overture to general open-source use
  18. Support for HTTP authentication filters in Java
  19. Built-in beer database to compete with Couchbase's substantial lead in the area
  20. Easier configuration of multiple TCP/IP adapters on a server and assignment of services to bind to them
  21. Full control over URL routing inside a database (e.g. "no legacy" mode where a Java class can map requests to resources)
  22. Keep up the ODS improvements - 8.x was a great set of releases on this front
  23. CalDAV and CardDAV support - Domino should be a close-to-drop-in replacement for Gmail on OS X and iOS
  24. Along the lines of #7, a general push to establish Domino as a top-flight NoSQL database and app-dev platform, not just "your old company's mail system"
  25. A real focus on standard-API performance, with less XSP-specific cheating

I'm sure this list is not exhaustive.

The Greatest Domino Poster of All Time

Apr 20, 2012, 7:23 PM

My company is moving out of the offices it has inhabited for... longer than I've been alive, basically. During this move, we've had plenty of opportunities to come across relics of its past form as an instructor-led training location: old courseware, ancient versions of Windows, NetWare, and Notes/Domino (from before it was Domino), and some priceless marketing materials. Some of the posters we've found are worth a good chuckle or two (a NotesMail poster boasting "Finally, a mail system you will never have to exchange"), but this one is my absolute favorite thing related to Domino:

Amazing Domino poster

It's amazing, like a Domino ad out of Yellow Submarine. What I wouldn't give to have IBM's marketing to be more "Then, suddenly, bursting from a magic lotus blossom" and less "The Social server software for Socially Socializing with Social people in a Social way - just ask this guy in a tie."

Fun fact: the hooded "executioner" guy in the top right being punched out by Domino is labeled "SuiteSpot". SuiteSpot, it appears, was some sort of Groupware/web/mail/directory offering from Netscape. I guess Domino won that battle.

File Under "Man, I Hope Designer Still Works After This"

Apr 8, 2012, 6:15 PM

Tags: domino

Well, I think the new blog has gone relatively smoothly, other than my accidental re-posting of my SQL-migration post (which oddly seems more popular than its first run, just one day earlier). That means it's time to get started on the next phases.

Other than the mundane setting-up-a-blog stuff like implementing search and proper draft posts, I have a lot of work to do surrounding my Ruby bindings. Since I'm going to eventually want Ruby "script libraries" and other handy non-inline uses, I'm going to need a proper Ruby editor. TextMate-via-WebDAV is OKAY, but not exactly elegant, and it involves too much overhead and awkwardness to be a great solution. Better would be to get DLTK - which has Ruby syntax highlighting - working in Designer.

On my first attempts, I ran into problems where using DLTK's update site demands an updated version of Eclipse, which could mean disaster for Designer. However, it turns out that GLTK has been part of mainline Eclipse for a while, so I figured I'd check to see if I could get to it via another route. Evidently, Designer 8.5.3 is based on Eclipse 3.4 (Ganymede), so I added Ganymede's update site to Designer's plugin installation list and, lo and behold, Ruby was there in the "Programming Languages" section. It still required the 3.4 version of org.eclipse.core, but, having backed up my Windows VM for this purpose, I felt daring. I clicked "Select Required" to select all applicable dependencies and hit Yes as many times as it took to finish the installation. After rebooting Designer, I found that Ruby scripts are now all fancily colored:

Ruby in Designer

If you'd like to try this yourself, I strongly suggest you do as I did and make a backup copy of your Windows VM first (or do whatever it is people who use Windows as their host OS do when they're about to ruin their system). Designer can be very picky about what plugins you install.

Making the Dogfooding Switch

Apr 7, 2012, 11:55 PM

Tags: domino blog

I've finally done it: I've switched my blog over to Domino. I did it for a couple reasons:

  • To silence the voice in the back of my head constantly saying "why are you using WordPress? You're a freaking web programmer! Write your own!"
  • To put my Ruby-in-XPages code through its paces in the way only a live site can.

I've already had to fix a couple holes in my Ruby adapter, mostly revolving around the fact that I haven't bothered to properly handle serialization and JSF's StateHolder interfact. For now, I've patched the worst problems, but this gives me a great reason to come up with a proper solution. To test it out, I'm making sure to avoid "#{javascript: ...}" code blocks in favor of using plain-EL and "#{ruby: ... }" exclusively. So far, the only really awkward parts are the giant swaths of yellow-squiggly "I don't understand this" underlining in Designer and having to put xp:repeat values on xp:dataContexts rather than writing the computation inline. Not too shabby.

The whole thing's a bit shoot-from-the-hip at the moment, since it's only existed for a day. There's no search (though that'll be easy), the site design is from my old college-era blog, and I have to write the posts' HTML by hand in the Notes client (you know, like on all modern blogging platforms). But hey, I have the old posts in there, plus an archives list and, as long as it doesn't mysteriously die again like it did a minute ago, Akismet-backed commenting.

So let's see how this thing works! Don't be surprised to see the WordPress version again if things go catastrophically wrong.

Import from SQL to NSF: It's So Easy!

Apr 6, 2012, 7:57 PM

Tags: domino sql

I decided I should probably finally get around to moving this blog from WordPress to Domino, if for no other reason than to have a perfect testbed for the weird stuff I've been doing lately. The first task is to write an importer, so I decided to just do a straight SQL rows -> Domino documents import. This couldn't be easier if you follow this simple guide:

  1. Write a Java agent that uses the appropriate JDBC connector for your database. In my case, it's MySQL, so I had it do a "show tables" to get a list of tables, then loop over those to do "select * from whatever" statements to get all documents.
  2. Since it's Domino, you can just do doc.replaceItemValue(column, rs.getObject(column))!
  3. Oh wait, you can't all the time. Make sure to handle a couple cases, like converting BigIntegers to longs and Timestamps to DateTimes.
  4. Wait, Domino doesn't even handle Boolean? For frack's sake, FINE: row.replaceItemValue(column, cell.equals(Boolean.TRUE) ? 1 : 0);
  5. Oh hey, that import went really smoothly! Now I'll just make some views based on the table name and... crap. Forgot to include the table name! Maybe I shouldn't have written this importer at four in the morning.
  6. Better delete all the documents and start over.
  7. What's this, an error when I went to delete all the documents in the All view? "Field is too large (32k) or..." oh no.
  8. Oh crap.
  9. Crap!
  10. Ah, I know - I'll just write an agent to do session.CurrentDatabase.AllDocuments.RemoveAll(True) and that'll fix that.
  11. Hmm, nope, that didn't work. Alright, based on the documents created around the same time, I can guess that it's the "option_value" field that's too big. Why did it even cram that data into summary if it completely breaks the document? Well, no point in dealing with that now. It's time for StampAll to take care of that!
  12. Nope? Okay, how about session.CurrentDatabase.GetView("All").AllEntries.StampAll?
  13. Not all categorized? What kind of error message is THAT?
  14. Time to delete the database and start over!
  15. Alright, THIS TIME, check to see if the String value is over, let's say, 16000 bytes and, if so, store it in a RichTextItem instead.
  16. Oh nice, it worked.
  17. Oh crap, I forgot to include the table name again.

And that's all there is to it!

#{ruby: 'it\'s a start'}

Apr 3, 2012, 7:55 PM

Tags: domino ruby

Oh man, I think this might actually work. Feeling adventurous this evening, I decided to look into the XPage runtime's expression language handler. After poring through tons of methods, interfaces, implementation classes, EXTENDED implementation classes, and disparate JARs, I narrowed the prefix handler down to the "FactoryLookup" property of the IBM-specific variant of facesContext's Application. With that, which is basically a hash, you map a prefix to a handler factory (it's always factories with Java, isn't it?). Once there's a handler registered, you can then bind expressions with that prefix.

This actually means what it implies:

<xp:text value="#{ruby: 'hi from ruby'}" />

And it works! Better still, the show-stopper in Designer seems to have been resolved in one of the recent versions: rather than error'ing out when it sees an unrecognized EL prefix, it nags you with a warning but otherwise proceeds without problem, saving the script as-is in the resultant Java.

Now, this is VERY much a first pass at the idea - all I did was write extremely skeletal implementations of the Factory and Binding classes and then I register them in the beforePageLoad event of a page:

var app = facesContext.getApplication()
var facts = app.getFactoryLookup()

var rfac = new mtc.ruby.RubyBindingFactory()
facts.setFactory(rfac.getPrefix(), rfac)

I can't even begin to stress how not the right way this is. It doesn't work in all controls (repeats, for example, seem to do an extra syntax check), it doesn't yet have any context from the surrounding environment, and, most importantly, who knows what horrible things it's doing to the HTTP stack?

Still, the crucial point is that it really, really looks like it can work.

The two classes I wrote, which no one should, under any circumstances, use, are here:

Putting the Domain Catalog to a Bit of Use

Apr 3, 2012, 1:33 PM

Tags: domino

Since I kind of backed my way into Domino development and administration, there are a number of areas of the server's functionality that I'm either unfamiliar with or casually brushed off as unreliable or not overly useful.

The Domain Catalog is one such area: I've been vaguely familiar with it, but have never bothered to tend to it or use it to solve problems. Fortunately, a problem it's perfectly suited to fell into my lap. In an overarching administration database, I want to get from a document with a database doclink to the database itself as quickly as possible. The easiest way, programmatically, is to use the .ServerHint and .DBReplicaID properties in a call to an unconnected NotesDatabase object's .openWithReplicaID(...) method.

This works well, but doesn't take into account the existence of local replicas - the admin database is running on one server, while the databases are created and linked to on the primary production server. Given that these servers aren't even in the same physical state, it's significantly faster to use a local replica when available, and only fall back when it hasn't been created yet. My first attempt at getting around this was direct: I first call .openByReplicaID(...) with a blank first parameter and, if it's not open, try the real server. Again, this worked, but is not ideal. It's programmatically ugly and, besides, there are more than just the two servers. .openWithFailover(...) looked promising, but seems to only accept real file paths, not replica IDs, which ruled it out for this purpose.

This is where the Domain Catalog comes in. It already contains a (hopefully-up-to-date) list of all of the replicas of every database in the domain along with everything I could need to connect to them. Furthermore, I realized I could use this information with a dash of extra intelligence: since I know which servers are physically closest, I could use that to find the "best" replica in each situation. So I made a view for this admin server ("Ganymede") of all of the database stubs, sorted first by replica ID and then by a column with this formula:

name := @Name([CN]; Server);
     name="Ganymede"; 1;
     name="Demeter"; 2;
     name="Invidia"; 3;
     name="Dionysus"; 4;

With replica ID in hand, this means that I can just do a @DbLookup() in the view and the first result will always be the best, regardless of which server it is. I have another column with the full "Server!!FilePath" path, so I can just pull that value and do it in one function. Nice, easy, and it lets Domino take care of the nitty-gritty details for me.

What Makes the Hassle Worthwhile

Apr 2, 2012, 10:34 PM

Tags: domino ruby

I've been toying with my Ruby servlet a bit this evening and it didn't take long to start having some fun. For example, here's a snippet from a page I'm building with Markaby, which is an aging little library that makes building HTML pages declaratively a cinch:

$database.views.sort { |a, b| <=> }.each do |view|
li { a } unless =~ /^\(.*\)$/

That prints out the names of all the non-hidden views in the database, sorted alphabetically, inside an HTML list. That's barely scratching the surface and not often useful, of course, but it's a proof of concept.. Server JavaScript can do some cool things, but the required syntax makes it a classless language by comparison.

I also realized earlier today that the same idea here can be translated to writing agents in Ruby by creating them as Java agents and putting the Ruby code in attached Resource files. It's a BIT of a hassle, in that I can't use the WebDAV trick to edit the scripts with TextMate, and it seems like I have to manually rebuild the agent whenever I change the script file, but that's not too bad. I'll probably try it out on some non-critical agents on my dev server to see if there are any critical memory issues.

Next step: figuring out this newfangled OSGi doohickey.

My Recurring Ruby/Domino Dream

Apr 1, 2012, 6:54 PM

Tags: domino ruby

As is no doubt clear by now, one of my obsessions when it comes to Domino is trying to make my programming life better, and one of the best ways I can think of accomplishing that is if I can make it so I can program in Ruby instead of one of the godforsaken languages natively supported.

In general, my attempts towards this goal have fallen into three categories:

  1. Accessing Domino from Ruby as one might a normal database. The very-much-in-progress fruit of this is my Domino API for Ruby project.
  2. Using Ruby as an alternative scripting language for XPages, like "#{ruby: blah blah blah}". This would be nice, but I haven't made any progress towards it.
  3. Running Ruby scripts out of an NSF via the web, as one might a non-scheduled agent

The last one has been the one at which I've taken the most swings, and I actually made some progress along those lines today. I decided to give a shot to using Domino's ancient servlet support combined with JRuby to accomplish this, and I think I've done it.

Since this is Domino, there were a couple hurdles:

  1. Servlets aren't automatically authenticated as the current Domino user. When I was looking into this, I read a lot about using NotesFactory.createSession(null, token), where "token" is the value of the LtpaToken cookie in an SSO configuration, which I use. However, this didn't work for me, possibly due to using site documents. Fortunately, there appears to be a better way: NotesFactory.createSession(null, req), where "req" is the HttpServletRequest object passed in to the servlet's method. This works splendidly, giving me a session using the current user without having to worry about figuring out their password or dealing with tokens manually.
  2. Finding the context database and script. The servlet is triggered properly by ".rb" extensions in file resources, but that still left the problem of actually opening that resource. Fortunately, the Session class has a resolve(String) method that can be used to get a Domino product object from a Notes URL. For file resources, this uselessly gives back a Form object, but you can get its UNID from the getURL() method.
  3. Reading the contents of the script. File resources appear to store their data in a $FileData rich text field, but that doesn't make it easy to deal with. For now, at least, I've gone the DXL route: I use the Document's generateXML() method, find the BASE64-encoded value in the <filedata> element, and decode that with sun.misc.BASE64Decoder.
  4. Actually executing the script. I banged my head against this for a long time: I kept getting NoClassDefFound exceptions for org.jruby.util.JRubyClassLoader when trying to run the script. I assumed at first that this was a problem with the class loader not finding this secondary class, so I tried tons of stuff with moving the JAR around, repackaging my servlet, and even trying manual class loading.  However, the exception was a red herring: it was really a permissions thing, so, not wanting to take any guff from a JVM, I granted myself all permissions and it worked great.
  5. Editing the Ruby script. This part wasn't really a crucial flaw, but the fact that Designer doesn't know about Ruby meant that editing the file resource involved no syntax highlighting or other fancy features. DLTK doesn't seem to play nicely with the Eclipse version in Designer and I've ruined my installation by messing with that stuff too much in the past, so I decided to try WebDAV. Unfortunately, Domino runs the servlet on WebDAV requests too, so, rather than trying to figure out how to handle that nicely, I just switched to accessing the database via a clustered server that doesn't run the servlet.

It was a lot of hassles, but it seems to work! The cleaned-up version of the method I used is:

  1. Put the "complete" JRuby JAR into Domino's jvm/lib/ext folder.
  2. Put the compiled class file for my servlet into Domino's data/domino/servlets folder.
  3. Grant all permissions to the JVM.
  4. Enable servlets for the server in question and specify "rb" as one of the file extensions.
  5. Enable the servlet in, as in my example.
  6. Restart HTTP/the server.
  7. Add .rb Ruby scripts as file resources to a database. I made one that makes a list of some documents from a view in my DB.
  8. Open the resource in a web browser and see it work! Hopefully!

One caveat: though I set the error and output streams of the Ruby environment to be the HTTP output, this seems to not always work, so I've found it best so far to use "$response.writer.println" instead of "puts" for writing text to the stream. I'll see if I can fix that. Turns out writing to the HTTP output should be done by calling setWriter(Writer) rather than setOutput(Writer).

Additionally, that "$response" there isn't a built-in feature of JRuby - it's a variable I put into the runtime. I do this along with "$request", "$session", and "$database". Think of them like the equivalent variables in XPages.

Some questions you may have are "is it stable?" and "is it fast?" and to those I have a simple answer: beats me! I just got it to work a bit ago and I don't know if I'll ever even use it. It's quite exciting, though, and that's what really matters.

In Between My Project and XPages

Mar 15, 2012, 4:11 PM

Tags: domino xpages

Despite my grousing about the state of programming for Domino in general and Designer in particular, I'm still mostly a fan of XPages. I use it for my guild's web site and pretty much every new project at work. However, I haven't been able to crack migrating my main work database template over.

Without getting too much into it, the point of the template is to create one database per project to act as a project web site listing online events with arbitrary registration forms and exit evaluations (among other things). Except for custom changes, everything is done via the normal Notes client, not Designer - pages exist as documents with a Body rich text field and associated data and register forms/exit evals, crucially, contain a set of response documents describing their fields. It's important to keep everything as visual as possible, because the users (the people at my company that set up these web sites) are not programmers.

I scrupulously try to avoid modifying the design of the database, so I go out of my way to do custom work via the existing mechanisms as much as possible, and I make pretty good use of rich-text-isms like embedded views, tabbed/computed tables, attachments, buttons, and computed text. In the case of the generated forms, fields can be placed either automatically (with a surrounding template of HTML per-field) or directly into the rich text body via <%FieldName%> placeholders (creating an experience like form design in Designer). Additionally, I have "WebQueryOpen" and "WebQuerySave" fields in the documents for formulas that I pass on into @Eval() in the equivalent places in the actual forms; most of the time, I use these for running agents.

Computed Text

Of the various potential show-stoppers, I think computed text fares the best. For one, it mostly worked - the computed formulas are indeed evaluated and the result is put into place correctly. However, they don't have the same environment you get with the "classic" design elements. The big one that I ran into immediately was @UrlQueryString(...) - it appears that the rich text renderer doesn't inform the rich text about its web environment completely. Prior to 8.5.3, "Display XPage instead" pages didn't know about the real URL, so they couldn't get query parameters at all, but 8.5.3 appears to pass that information along properly. So that means it MIGHT be fixable, if I find a way to properly set fields in the document before the rich text is rendered, so I can set QUERY_STRING - if I can do that, either @UrlQueryString(...) will work or I can manually parse the string as needed.

Embedded Views

They work! ...ish. It looks like icon columns get their URLs a bit messed up, but I can't think of a time when I actually used them, particularly in a situation where I couldn't just write out the URL myself.

Tabbed/Computed Tables

From a cursory glance, I think I'd be SOL when it comes to these. Tabbed tables render flattened out and computed tables don't seem to work. The former can be improved on easily with Dojo, but the latter would mean replacing server-side business logic with client-side JavaScript, which would lead to headaches.


These show up as "(See attached file: foo.jpg)". So... not functional. I might be able to fix it by using a filter on the text to replace the HTML with a link to the document, but I don't know if I could get the attachment image properly. Attachment images aren't amazing, but sometimes they do the job.




In some cases, I could run the formulas through session.evaluate(), though I'm not even sure queryOpenDocument/postOpenDocument let me hook into the right spots (sometimes I set fields on document open). I don't think that would run agents, though, so I'd be stuck trying to parse out the text to look for @Command([ToolsRunMacro]; "...").

User-generated Forms

This is the toughest one. I've given this thought a number of times, and I can't think of a great solution. I can't just re-use the subforms I have already, I don't think I can generate and import an XPage via DXL (though I haven't given that a significant shot), and I can't just do a <xp:repeat/> to create all the fields, since some are in the body area. The main routes I can think of to try are a) arranging the fields on the page after the fact via JavaScript and b) generating the page components on the fly with Java or Server JavaScript. I'll probably give the latter a shot eventually, but I expect it to be a bag of hurt.


It's a lot of hurdles! I'd love to switch over to XPages, particularly since, for everything that's more difficult than using classic elements, there are 10 things that are way easier. It'd just be quite an investment of time to merely get up to par with the functionality I already have, if it's even possible in all cases.

Quality of Life: Eclipse Color Themes

Mar 10, 2012, 6:21 PM

Tags: domino

Well, this is a nice quality-of-life improvement: the plugin Eclipse Color Themes, which (so far) works just fine in Designer (8.5.3). Having to go through every single editor type and manually pick each color for each code element every time I wanted to change a color theme was always a huge annoyance, particularly compared to other editors. Fortunately, that plugin handles it pretty well, though it unsurprisingly doesn't support LotusScript, so that's still manual. I installed it without problem, found its settings in General\Appearance\Color Themes, installed a port of my preferred theme, and saved myself tons of hassle.

How I Want To Use Domino

Mar 7, 2012, 12:43 PM

Tags: domino
  1. How I Want To Use Domino
  2. How I Want To Use Domino, Take 2

From my perspective, there are three main problems with Domino: the limits, the client, and the server. Now, that's a lot of stuff... most of the product, in fact. However, the facts that I'm still programming for it and that my company's 16GB project-tracking database is as snappy as it was when it was empty attest to the core quality of the product. Off the top of my head, I can think of a number of things that make Domino salvageable:

  • Reader fields. These are hard to beat and hard to find elsewhere. While you can do everything without them that you can do with them, you'd likely spend a lot of time worrying about edge cases or limiting yourself to coarse security. If a user doesn't have Reader access to a document, it doesn't exist (well, except for some shadows in view indexes).
  • Solid, straightforward directory integration. Going along with reader fields, it's very helpful that the directory system is part of the overall product and works well enough with LDAP that you don't have to worry too much about it. User names, group memberships, and roles all make sense.
  • Full-text searching. It works and it's quick.
  • View indexing, more or less. As long as you don't rock the boat too much and don't think about what you used to do with SQL tables and views, Domino does a fine job of maintaining views for you.
  • File attachments. They work well and can even be full-text searched if they're the right type.
  • ID files. I used to hate these, and they're often more hassle than they're worth (as my C API programming is demonstrating), but they have some distinct advantages. If you squint, you can see them as really long password that may themselves have another level of password. They're also essentially the same concept as SSH keys. They also tie into handy encryption and certificate abilities that I don't usually have much use for.
  • Potential capability as a blob store. I saw a great post a while ago talking about the ability to just cram arbitrary data into a Domino document, and I think this is a very valuable line of thinking. If your data fits into the model of "a couple queryable bits and then a block of arbitrary data" or the "upside-down" method of memory-storage-first from the post, such a setup could work extraordinarily well.
  • Replication and clustering. Sync is not hard... unless you let your deletion stubs expire.
  • Agents, more or less. The language choices and lack of immediacy make agents kind of a pain sometimes, but it's still handy to have code directly associated with the database with schedule and event triggers.
  • Read marks. It's nice to have the option to let the server handle this for you when it fits your needs.

That's on top of the various benefits you get from document databases generally, like multi-value fields and the lack of schemas. Note, though, that none of these are (directly) related to Domino's chops as a mail platform, web server, or GUI app environment, its choice in programming languages, the value of its standard array of templates (though they can be handy), or pretty much any virtue extolled by its marketing materials. The guy writing on the invisible whiteboard on IBM's page doesn't care about reader fields or blob storage.

Though most of my Domino work is for the web and XPages are a better way to do that than the legacy elements, it's still a drag. Java as a language is a hassle (the platform has its charms), Server JavaScript isn't a full replacement in the way that Groovy and JRuby can be, and working on top of a giant Jenga tower of Java classes and abstractions can be hazardous to your health.

I really just want to treat Domino like MongoDB, CouchDB/Couchbase, and the rest: an environment-independent document database. This is probably not worth the effort, particularly since IBM seems passively hostile to the notion, but I find it to be a compelling idea. It doesn't have to fit into, say, Rails, but grinding Domino down to its solid NoSQL core could open up a lot of possibilities.

Some Niceties of Implementing a Notes API

Feb 20, 2012, 7:56 AM

Tags: domino ruby

There are a couple things about writing my Ruby wrapper for the C API that make it particularly fun, mostly related to getting to add abilities that I desperately wish were there in the normal APIs.

  1. Ruby-style (forall) looping. Anyone who has iterated over a NotesDocumentCollection knows the drill: set a variable to the first element, start a while loop, and make sure to set the variable to the next one at the end. Writing it one time isn't so bad. Writing it hundreds of times, though? It gets to be a drag. Getting to write docs.each { |doc| ... } is a breath of fresh air.
  2. Easier design-elements-as-notes access. The normal API lets you get a NotesDocument version of a NotesView via its UniversalID property, but for everything else you need a NotesNoteCollection, which is a hassle. Since all design elements are Documents anyway, I've just made their wrapper objects subclasses of Document (though I may change that to just a #document method if it gets hairy) and I've put a #get_design_note method on Database that lets you find a design note by name and flag class (NIFFindDesignNoteExt).
  3. HTML and DXL everywhere. I use DXL fairly constantly (mostly for design elements), and for the most part it's a hassle. Not only do you have to create a NotesDXLExporter, but you also have to get the Document version of the design note you're dealing with and run through its process. Not impossible, but sort of a hassle. Now, though, I just put a #to_dxl method on everything - this is like the .generateXML() method in Java, but consistently applied. Similarly, I can't count the number of times when it would have been handy to get an HTML representation of some element, even if it was just the dated stuff that the legacy renderer puts out. Since that's all there in the C API, I just put a #to_html method on everything and a #get_item_html method on Document for very easy access to web-friendly versions of MIME and Rich Text items.

It's just a shame that I probably won't be able to use this: what I'd really want to do would be to use it like a database driver on a Ruby-driven web site, but the fact that it's so tied to ID files makes that tough. Still, it's great exercise to write it, and maybe I'll cave and set up some sort of multi-process hydra beast to suit my needs.

Started Work on a Ruby Wrapper for the C API

Feb 15, 2012, 7:28 PM

Tags: ruby domino

As I had mentioned before, I've been tinkering about with the Domino C API, specifically with Ruby. Although I'm not sure I'll actually have a use for it (from what I can tell, the C API is very tied to ID files and threads, which would make a multi-user web server thing cumbersome), I've decided to go for it and write a wrapper for the API generally based on the Java/LotusScript API. This is serving a number of purposes:

  1. It gets me off my (metaphorical) duff and in front of a text editor during my off hours.
  2. It lets me work with Ruby in a real capacity. And moreover, it'll expose me to how to write a Ruby library.
  3. It's introducing me to proper version control, something I absolutely need to do better.
  4. It will eventually force me to write structured documentation, which I haven't had to do in the past.
  5. It should fill a hole currently only partially filled with OLE-based libraries, though the ID-file thing will make it awkward.
  6. It brings me back to the world of C structures and pointers, albeit cushioned by FFI.
  7. By the time I'm done, I'll know Domino inside and out.

So far, I have some of the basics working: creating a "Session" by passing in the program directory and the path to an INI file, fetching databases, traversing and full-text-searching views, reading view entries, and reading basic and simple MIME data from documents. Obviously, there's plenty more work to do, and I don't know if I'll bother implementing the more esoteric classes like NotesRegistration, but it's off to a fun start so far:

Point 7 on that list has proven surprisingly interesting. It's fun to see the bits that apparently never fully made it onto the product (like number ranges), the little optimized paths where the documentation goes out of its way to point out that it's particularly efficient, and all the data structure sizes that create the various limitations that make Domino programming such an... experience.

This Dynamic View Customizer Is Getting Into Shape

Feb 13, 2012, 10:08 AM

Since last week, I've made two nice improvements to my dynamic view customizer:

  1. I added some support for twistie images when the referenced DB is on the same server. The code assumes that the referenced images are image wells with at least two entries, but I can't imagine why that wouldn't be the case in practice.
  2. I vastly improved my handling of color columns. Previously, I had been resorting to hacky methods like hidden <div>s read by JavaScript or surrounding <div>s styled to take up the whole cell, but those were terrible and easy to break. Now, though, I'm doing it right: the code adds a value binding for the column's style attribute to create the CSS for each cell, which is ideal.
  3. Empty categories now are translated to "(Not Categorized)", as in the client.

Now that my code is presentable (albeit oddly structured and uncommented), I figured I may as well toss it up on GitHub:

To note if you want to use this in your own project: it references the "mcl.JSFUtil" object, which started as the mindoo object of the same name and has since turned into my bin for common functions. The methods used here are getSession(), which just gets the value of the JSF "session" variable, and the xmlEncode() and specialTextDecode() functions from my string utils. Additionally, it references "com.raidomatic.xml.*", which are quick wrapper classes I made to ease basic XML access, and which are also available on GitHub:

Enhancing xe:dynamicViewPanel For My Own Purposes

Feb 9, 2012, 8:45 AM

I think I have my view rendering problem licked. To recap, I've been working on a way to show views in XPages that met a couple requirements:

  1. Entirely dynamic. Since this will be for a combined reporting site that will show views from customized project databases of wildly varying needs, I couldn't make any assumptions about view layout, categorization, or content. It should pull as much display information from the view design as possible.
  2. Fast. Some of these views have thousands of rows, so it should be as fast as possible to load from the database and render for the browser.
  3. Pagination. Though I don't really like pagination for the average case (who wants to look at a 40-row view 30 rows at a time?), sending thousands of table rows to a browser causes a miserable experience even when that browser isn't IE.
  4. Full-text searching and other Domino goodies. I don't want it to be a pain to just be able to search through a view like you would in the Notes client.
  5. Support for "second tier" Domino view features. I use special text, column hide-when formulas, "show values in this column as links", Notes-style [<span>pass-through-HTML<span>], and color columns constantly.

Of these, 1 and 5 are the most important.

I originally wrote my own code to generate a big HTML blob for the view, which was perfect on points 1 and 5 (and looked great, since I converted categories to OneUI sections), but wasn't as hot on the other points. The Extension Library's xe:dynamicViewPanel control hits points 1 through 4 with aplomb, but misses the crucial point 5. For one crazy moment, I toyed with the idea of trying to get the output of the "legacy" HTML view renderer (which is perfect on all points) via client-side JavaScript or some other hack, but decided against it as being too rickety.

I considered changing my custom HTML code to instead create a bunch of Java objects to make pagination easier, but performance would be an issue - it's tough to beat built-in controls for raw speed. As anyone who has looked at the code for the mail template knows, IBM/Lotus cheats like crazy, and often the only thing you can do is to just go with what they've done and beat it into shape. Thus, the answer came to me: customizer beans for the ExtLib's dynamic view panel.

The dynamic view panel has a "customizerBean" property that takes a string class name of a bean to create to hook into a couple overridable methods and events:

  • ViewFactory getViewFactory() - this lets you override how the panel pulls in all of its information about the view.
  • boolean createColumns(FacesContext context, UIDynamicViewPanel panel, ViewFactory f) - this lets you override how the panel uses the gathered view information to generate the list of data table columns (or simply run code before that happens, if it returns false)
  • IControl createColumn(FacesContext context, UIDynamicViewPanel panel, int index, ColumnDef colDef) - similarly, this lets you change the way each individual column is created based on the ColumnDef (an object generated by your ViewFactory)
  • afterCreateColumn(FacesContext context, int index, ColumnDef colDef, IControl column) - this is an event fired after each column is generated, allowing you to customize the generated column without having to build it from scratch
  • afterCreateColumns(FacesContext context, UIDynamicPanel panel) - similarly, this lets you run code after the column creation is done

These events provided just the hooks I needed to get the dynamic panel to do what I wanted.

Special Text

To add special text support, I added an implementation of afterCreateColumn(...) that replaced the column's default ViewColumnConverter with my own subclass that overrides getValueAsString(...). With that, I can walk up the current component's ancestors to find the panel, find the individual ViewEntry's variable name from there, and use that to process the special text properly.

Column Hide-When Formulas

These are a bit trickier. The getViewFactory() hook lets me provide my own subclass of DefaultViewFactory, but there's a reason that the dynamic view panel doesn't support these formulas by default: they're not exposed by the NotesViewColumn class. The two ways I can think of to get at this information are the C API (or otherwise reading the design information from the view note directly) and DXL. The former is no doubt significantly faster, but DXL is much, much easier and is good enough for my needs at the moment. So I export the view as DXL and process its column nodes and their children. Then, for each with a hide-when formula, I evaluate the formula in the context of a new document in the project's database and use the result to set the column-hidden flag.

"Show values in this column as links"

This goes hand-in-hand with the hide-when formulas. The column nodes in the DXL have a showaslinks attribute that I can look for to set this flag.


Much like special text, once I have the custom converter attached to the column, I can break apart and reassemble text values on [< and >] and XML-encode the non-HTML bits. While I'm attaching the converter, I set the column's content type to "html", since I'll be handling the encoding.

Color Columns

These are... tricky, and I'm not yet proud of my fix. There are two problems: getting each cell to know about its color information from previous cells and then actually applying that color information. For the former problem, I dealt with it by adding an "activeColorColumn" property to the ColumnDef object and setting it to the programmatic name of the most recent color column before each column. The latter is the ugly part. You can't just set the column's style, since that will apply to the entire column and make a fine mess of everything. What you really want to do is assign it to the current <td> element, but I don't know a good way to do that. In the mean time, I've hacked around it by making a hidden <div> in each cell with the style tucked into a data-cell-style attribute. Then, I have some client JavaScript look for these divs and apply their styling to the parent table cell. It's not pretty, but it works for now.

If I get that last bit of code cleaned up to a point where I'm comfortable letting people see it, I'll post my example code. In the mean time, I'll count this whole thing as a win for only writing the code you need to and letting the platform take care of the rest.

Formatting View Content on the Web

Feb 1, 2012, 10:06 AM

Tags: domino

My current work project involves displaying arbitrary views from various databases into a combined reporting site (written in XPages). This has presented me with two hurdles: pulling in the data completely and accurately and then figuring out a good way to format it.

The former problem is one of the few areas where it seems like "classic" Domino development has an edge: views render rapidly and, as long as you've configured the server to display lots of rows or add in pager controls, completely. And with the recent "enhanced HTML" property, they generally get tagged with some useful CSS classes. The default XPages xp:viewPanel control, by contrast, needs the column definitions at design time, which makes it tough to work with. You can sort of roll your own with nested xp:repeat controls, but you quickly run into problems with categories and other Notes-y features. Fortunately, the Extension Library, as usual, comes to the rescue: the  xe:dynamicViewPanel does pretty much what you want: you give it a view and it renders it out nicely. Using it, you don't have to worry about handling categories, you can add in standard pagers, and you can include all the standard filtering and searching options on the view. It's not perfect (yet), though: it doesn't handle fixed-value columns (like a formula of "hi"), use color-column information, follow column hide-when formulas (though it supports the static "hide this column" checkbox), do Notes-style bracketed pass-through-HTML, process special text, or honor "show values in this column as links." These are all understandable limitations, being as they are either rarely encountered or technically difficult. Unfortunately, I've run into all of them rapidly, so I don't know if I can use xe:dynamicViewPanel as-is. The best thing to do would be for me to roll up my sleeves, implement these features in it, and submit my changes, and maybe I'll do that down the line, but there'd be a lot of learning overhead before that point. What I've taken to in the mean time is a Java class to write out the view contents as HTML. I add the color-column info via inline style="..." attributes, look up the hide-whens by exporting the DXL and then running them through session.evaluate (read: not fast), and split and recombine the strings to handle pass-through-HTML. Unfortunately, I lose speed (because I'm presumably not as good at this as IBM's programmers) and the flexibility to easily use searching, filtering, and paging. The first two will be relatively easy to implement, but paging will be a problem, since the XPage just gets back a big string blob. I could return Java data structures, but then I'd run into problems with categories. I could write classes to dynamically add XSP elements to the page, but at that point I'd be best off cracking open the ExtLib source code anyway.

The other problem is that, regardless of how I render the view, it can be tough to actually display all the data well on a web page. For normal views, with a handful of columns, it's fine to just plop it right on the page and be done with it. However, it doesn't take much to make the table look like crap: headers that are way too long, large strings in the row data, or just having lots of rows all very quickly stretch or break the design. I think I'll try to handle the headers by adding some CSS text-overflow rules to keep them to a fixed width, which will also mean adding in a title="" attribute so the user can hover over them to see the full value. Still, not a big problem. Large data cell values are a bit tougher. Theoretically, I could do the same thing: have the data overflow and add a tooltip. That would work fine if it's a rare occurrence, but what if the whole point of the view is to be able to easily look at that data. I may try to pull out the row-height property from the view and use that to provide a max-height to each cell via inline CSS, so that way I could solve the problem on a view-by-view basis. I could consider doing the same with width as well, rather than going the default route of letting the content width determine the column width. Having a lot of rows can be partially solved with pagination (unless I'm going the "dump out a blob of HTML" route), but is that really a good user experience if the goal is to see all of the data? Paging on the web is pretty awkward, with the number of rows never exactly matching the height of the viewport, worrying about scrolling back to the top when going to the next page, and properly dealing with categories. Via CSS, I've made it so that the primary content area has its own scrollbars via absolute positioning and overflow: auto and that works pretty well for the most part, but I still run into trouble with views containing thousands of rows: with a lot of data, it simply takes a long time to download the HTML and render the table. There are a couple options provided by Dojo and the Extension Library that provide fixed-size grids to store data that I've tinkered around a bit, and maybe I'll settle on one of them, but in the mean time every solution has some serious problem that I quickly run into with my example data.

Getting My Feet Wet With Ruby and the Domino C API

Jan 26, 2012, 4:07 PM

Tags: domino

My search for a useful way to use Domino as a database back-end for a Ruby front-end has continued and, more specifically, has continued to be difficult.

I gave a shot to the Java CORBA API. Initially, that went well: after grabbing NCSO.jar from my Designer installation, enabling DIIOP on the server, and setting up JRuby, I was able to connect to the server using the host name, canonical user name, and password. Great! However, once I exported my app's Java model classes (modified to use NotesFactory instead of the JSF runtime), I immediately ran into a problem: NotesView.resortView() is not implemented in CORBA. Guh. Theoretically, I could rework the app to use a different view for every index, but it's a canary in the coal mine: no doubt plenty of other new things (FTSearchSorted, for example) have been left unimplemented as well. Plus, this is Domino, so "new" probably refers to anything added after, say, R5.

So that's out, at least for now. That leaves me with the C API. Oh, the C API... so much promise, such a PITA to use. I haven't used C itself much since college, and my experience with the C API is limited to progress bars in the Notes client and, more usefully, my authentication filter for my server. However, I'll be programming in Ruby, not C directly, so it won't be so bad, right?

There are three main ways to work with C libraries in Ruby, I've learned: writing a Ruby module, DL, and FFI. In the first case, I'd basically have to write my API in C and then just use those objects from Ruby. That would be the best performance, but it's just asking for a nightmare. DL is programmed in pure Ruby, but it's so sketchy that just loading the library causes your program to emit a warning telling you you probably shouldn't be using it. FFI, though, is pretty great. Much like DL (or declaring external functions in LotusScript), you declare which functions you want to import, along with their signatures, and then it more or less just works.

That "more or less" is, naturally, tricky. I ran into tremendous trouble just getting it to load up, but most of that was due to using a 64-bit OS and a 32-bit Domino installation. I may make a post about it some time in case anyone else is crazy enough to try the same thing, but the upshot is that I ended up compiling my own 32-bit Ruby environment, which started to work.

Since then, I've been wrestling with the inherent traits and idioms of C (using string buffer arguments to return values instead of just returning a "string", for example) and the specifics of Notes (needing to be pointed to the executable directory as well as needing an ID file, an INI file, and a names.nsf to find the servers). After thrashing around for a while, though, I have it working to the point where I can point the initialization routine to an arbitrary INI file and it can pick up on everything it needs from there (I'm using a password-less ID file - I don't know if you can automate passwords).

Now that I have my foot in the door, it's "just" a matter of learning the entire API. I'm looking forward to that part, though - I've demonstrated that I can open a database on a server and get information out of it, so it seems like I have the right environment. If all goes well, I should be able to create a proper Ruby API for Domino, and a very fast one at that.


For reference, here's the script I've written. Don't expect clean code, reusability, comments, or proper methodology - this file is the result of random mauling during the course of development and just barely works to demonstrate the idea. It DOES demonstrate it, though, and works to ping a (hard-coded) server and get some info about its names database.


A Couple Handy Domino Java String Utils

Jan 25, 2012, 11:08 AM

As most developers probably do, I have a grab bag of "utility" functions/methods I use across various projects, and I figured I'd post a few of the handier ones.

The first is a basic XML-encoding function that just takes a string and returns something suitable for putting into an XML or HTML file. I've been too lazy to figure out if the standard Java library has an equivalent that doesn't involve creating an actual XML document in memory, so I just took the one I use for LotusScript and ported it over. It passes standard letter and number characters through as-is and then just encodes all others as Unicode entities. The string size increases accordingly, but it's presumably fast and it gets the job done, even if you have oddball characters.

The next set are just quick-and-dirty Java equivalents to the StrLeft, StrRight, etc. functions from LotusScript. They do more or less the same thing as in LS and make it so you don't have to worry about string indexes all the time.

The last is potentially the most useful, since it's the most esoteric: a special-text decoder. I've had a couple occasions where I want to spit out the contents of a view for the web, but also want to preserve things like child counts on categories. Fortunately, special text is stored in the view column in a workable format: a special non-alphanumeric character, a single letter indicating which function it corresponds to, a number representing the parameter count for the function, and then a delimited list of those parameters with character counts. Using that, I wrote a routine to process all of the special text functions I could think of other than @IsExpandable (since that's meant for the Notes UI). It even reproduces the weirdo behavior you get when you pass a multi-character string to @DocLevel.

Confound It; That's Two APIs Down

Jan 22, 2012, 11:02 PM

Tags: domino

In the interests of getting crap done, as soon as I was finished with my previous post, I fired up TextMate and a couple Terminal windows to start writing a Ruby wrapper for the DAS API.

It started out great! Wanting to avoid the minor hassle I ran into before with Ruby's built-in Net::HTTP library, I did a quick search for Ruby HTTP/REST libraries and picked one that worked well, named HTTParty. Before long, I had the rudimentary elements of a Notes API working, with classes to represent the server connection, NotesDatabase, and NotesView.

I was about to start working on the NotesViewEntry class when I decided I'll need a way to page through the view, since the call to the view "collection" (wisely) doesn't return all of the data at once. The API developers cleverly store this information in an HTTP header:

Content-Range: items 0-9/35

Perfect! But, um... I was getting the list of available forums as Anonymous, which can only see two, not all 35. Sure enough, the results only contained the two Reader-visible entries (phew), but that left a showstopping problem: I have fairly frequent need for the count of entries, and I don't relish the notion of having to fetch all of the data to get that. Crap.

So DAS is out for now. I remembered, though, about ?ReadViewEntries, an older and thus clearly more mature data-access API. I could probably do everything view-related I need with that and then use DAS to get any non-view document data. So I hit the view with ?ReadViewEntries&OutputFormat=JSON (the results are the same with the XML version) and, lo and behold, the top-level object fields (with data snipped) revealed the same lack of Reader knowledge:

{ "@timestamp": "20120123T034916,85Z", "@toplevelentries": "35" }


My quest to treat Domino like a standalone database is off to a rocky start. I can think of a few remaining options:

  1. COM: I don't deploy on Windows, so... that's out.
  2. C API: seems like overkill and just asking for all sorts of new bugs.
  3. Web Services: MIGHT work, but I can't imagine the performance would be anywhere near acceptable.
  4. Some sort of crazy thing with servlets: seems too crazy.
  5. JRuby and the Domino Java CORBA API.

I think the last is the most plausible. My forum app already has Java classes in between the XPages front-end and the Domino database, so I could possibly take those wholesale and just change the methods I use to get the session and current database to instead initialize a CORBA session. I'm a bit nervous about speed and running across some of the various "not implemented in CORBA" landmines hidden across the API, but it might just work. If it does, my next task will be deciding on a web server and a Ruby web framework that's not terribly SQL-centric.

Keychain DB: Very Rough First Script

Jan 15, 2012, 4:20 PM

Tags: domino

I've had a chance to start on my Keychain project from last week, enough to put together a thoroughly rough and unmaintainable script to do the uploading. It has all kinds of horrible properties: it doesn't do the Keychain dump itself (instead reading from a hard-coded file containing a keychain dump from the "security" tool), it doesn't abstract away any of the Domino DAS access, it doesn't check for existing versions of the items, it doesn't handle field data types properly, and it even writes to the filesystem. But hey, it's a start:

keychain-upload.rb (requires the json rubygem and uses auth information from ~/.netrc)

In spite of all its faults, it DOES push the data up to the database, and that's something. It stores the keychain file name in a field named "keychain", the "class" in one called "entry_class", the data in "data", and the rest of the arbitrary fields from the item in fields prefixed with "attr_". Once it was in the database, I set up a view called "Passwords" with the following formula:

SELECT Form="Item" & entry_class="genp" & attr_type != "\"note\""

The extra quotes around the type field's value are an artifact of the non-existent type handling in the original script - it just stores the values exactly as they appear in the dump file, rather than converting them to null, numbers, or strings.

It's not pretty, but I now have a way to view my Keychain entries on the web. I'll probably give a shot to doing this the "right" way via a Cocoa app and structured classes down the line, but for now it'll already be useful to me.

A New Personal Project With Keychain and DAS

Jan 11, 2012, 7:18 PM

Tags: domino

With Apple's transition to iCloud, they're getting rid of Keychain sync. This is too bad, since that was one of the MobileMe features I actually used, loved, and never had problems with. Fortunately, all is not lost: worse comes to worse, I can pick up a copy of 1Password, which has the advantage over MobileMe of being cross-platform.

But I'm a programmer, right? Why buy something - especially for $50+ - when I can just write something myself? My needs at the moment are fairly simple - I only REALLY want a way to back up my Keychain and view it on the web, for when I'm in Windows or on my phone and want to check a password. I only use the one Mac currently, so I don't need proper sync, and I don't need to pull new passwords back down into the Keychain. Maybe down the line, but one step at a time.

The Keychain itself takes the form of a file housing a collection of records with a body of essentially arbitrary data (usually a string, but it can be different for certificates or notes), with a composite key consisting of, I think, the type, server, account, and maybe some other fields. The only thing that would make this a better test case for Domino 8.5.3's shiny new Data Access Services would be a 32-digit hexadecimal unique identifier, but it's pretty darn close as it is.

The main problem I have to getting starting is actually getting at the data. I think there are two main ways: the C-based security/Keychain API or using the "security" command-line tool, which is a more-or-less friendly wrapper for much of the same functionality. Being the lazy type that I am that only deals with C when I have to, I'm starting with the "security" tool. It has a handy "dump-keychain" command, so I can type something like "security dump-keychain login.keychain" and get a formatted list of all entries in that keychain, minus the "data" field. I can add the "-d" switch after "dump-keychain" to include that field, but that comes with a big caveat: you have to click an "Allow" button for every single item. Given that I have about 1,100 items, I'd rather avoid this fate. I've tried running it through sudo, but to no avail. I've tried fiddling with the "authorize" command in the tool, but also with no luck - I'm not sure if I'm just doing it wrong or if it's related to something else, which may or may not be what I want. I could use the "-r" switch instead of "-d" to get the encrypted data, which would satisfy my backup desire, but not my "check it from the web" desire. Absolute worst case, I could put on a movie or podcast and click "Always Allow" a thousand times.

However, once I get past that hurdle - whether it be via the C API or a lot of clicking - the rest should be easy. I should be able to create a basic database, grant DAS access to it, create a view sorted by the various components of the items' composite keys, and do a basic "view lookup, then update or create the doc" routine, pushing up the field values pretty much as-is. That should act as a perfect basic-case project for trying DAS out while also solving an actual problem I have.

Pondering RSS Syncing

Nov 23, 2011, 1:10 PM

Tags: domino

I was listening to the latest episode of Build and Analyze on the way home from work yesterday and, as I am wont to do, I started yelling at my iPhone when they started talking about Google Reader and the difficulty of syncing. Admittedly, at the end, they got to the fact that, even if you could do it technically, it'd be tough to make money off of providing an RSS sync server. That part is fair enough, but I still can't let the technical difficulties stand, and I've been thinking more about how it would be done in Domino.

In the basic form, the problem in question is pretty much exactly what NSF and the Notes/Domino relationship is designed to do: seamless replication, deletion stubs, unread marks, and so forth. In fact, RSS syncing is a better fit for the model than mail, since mail required adding all kinds of extra (but useful) functionality, while RSS syncing would just be data and an agent to fetch the feeds periodically.

The way I figure it, there would only be a couple technical hurdles, both related to scaling: storing large volumes of data and fetching new feed content periodically.

Storing large volumes of data might not be too bad. There are a couple ways you could do it. One would be to store the user's list of subscribed feeds and "read" stubs in one database per user, and then store the feeds and feed content in another database (or databases), and do all data access via agents or web services that would pull the data from each distinct location. Another way could be to store the feeds and entries in each user's database, keeping the feed content as attached HTML documents and letting DAOS handle efficient storage. The latter route would let you take advantage of the Domino Data Service and read marks (which DAS conveniently supports).

Updating the feeds would be rough for a single server, but the job could be farmed out to many clusters in a server. You could write agents that would determine the server they're on and, based on, say, its name, pick a slice of the feeds to check, so if you have 10 servers, each would update 10% of the feeds.

I'm sure there'd be other roadblocks during actual implementation (it IS Domino, after all), but I think that'd be basically all you'd need on the server side. The client would be a little tougher, since you couldn't just use NSF and selective replication, but that wouldn't be terribly difficult to handle.

It's too bad it's likely not profitable, between licensing, hosting, and bandwidth costs - it'd be a fun project to try out.

That Counts as Progress

Nov 22, 2011, 4:28 PM

Tags: domino xpages

A while back, I described the problem I'm having in my guild-forums XPages app, which is that it very easily gets its environment out of whack, to the point where changing any design or data note from outside the XPages environment caused an "X is incompatible with X" ClastCastException. This improved gradually over time. At some point in 8.5.2, I started being able to modify data documents again without the problem. When 8.5.3 came out, it improved again: I can now replicate over changes from my development server to the production one without having to bounce HTTP on the production one. At that point, it became much less of a hassle - sure, I still had to re-save a Java class file every time I modified an XPage, but that was only in development, so I could deal.

However, I think I've found the proper solution. In 8.5.3 (I think), IBM added a custom editor for the file (which you can reach using the "Package Explorer" Eclipse view, in whatever.nsf/WebContent/WEB-INF). Many of the options are duplicates of what you see in the normal "Application Properties" editor (since they edit the same properties), but there are some nifty additional goodies. I won't go into them all, but I urge you to take a look. The important one here is in the "Timeouts" section of the "General" tab: "Refresh entire application when design changes" . That sounded perfect and, lo and behold, it seems to do what I want: once I turned that on, re-saving an XPage stopped causing the ClassCastException. I haven't given it a proper testing, so it might not be everything I'm looking for, but I'm thrilled at the initial results. Having to re-save a Java file for each change wasn't a HUGE problem, but it was a niggling one, and not having to jump through the hoop helps make the whole environment seem less rickety.

That's Weird

Nov 9, 2011, 9:35 AM

Tags: xpages domino

Yesterday, I started working on a small sidebar widget app using an XPage, after finding out that XPages can now (as of 8.5.3) be used in the Notes sidebar in the same way that Forms could before. It's quite a simple page, very Twitter-like: one text input field and then a list of posts. However, even though it's very simple, I ran into two annoying bugs quickly.

The first of them is a Schrödinbug. I set up some code in the "onClientLoad" event to start a setInterval to do a partialRefreshGet on the list of posts, so it would keep itself up to date:

<xp:eventHandler event="onClientLoad" submit="false"> <xp:this.script><![CDATA[ setInterval(function() { XSP.partialRefreshGet("#{id:messagesPanel}") }, 2000) ]]></xp:this.script> </xp:eventHandler>

Simple, right? But I started getting errors in the status bar, along the lines of somethingorother not being defined. That's... odd. So I swapped over to Safari, figuring it'd be some XPiNC-specific thing, but I started getting similar (albeit more specific) errors there, along the lines of:

TypeError: 'undefined' is not an object (evaluating '_166.formId')

Huh. So I tracked it down and the problem was in the code called by partialRefreshGet, where "_166" is the name given to the second (optional) parameter. It makes sense when seeing it - though the code is supposed to fail over when it doesn't work, I can see why the browser would throw up its hands when you try to get a property of an un-provided parameter. But this has worked before! Just looking around on the web, you run into plenty of examples that leave out the second parameter. But sure enough, when I changed the line to XSP.partialRefreshGet("#{id:messagesPanel}", {}), it started working great. I can only think that this is either some 8.5.3-specific bug or some weird thing I managed to do in my code... but there's not even really enough code to mess up.

The second bug is still annoying, and it's somewhere in between a bug and a security feature of Firefox/Gecko. Basically, while you can call the .click() method on a button in client-side JavaScript in Gecko, it doesn't behave exactly like clicking on the button. Specifically, I have a CSS-hidden button that executes the actual action of creating the message document from the text you type in, and I want it to happen when you hit enter. The button itself works great - if I have it show up, I can type, click, and it executes the action and partial-refreshes the list of posts. However, if I hit enter, which uses the .click() method, I get an error about not being able to refresh that part of the page, but then it forces a full-page refresh and still works. So it's KIND OF clicking it, but not quite.

I'm not sure what to do about this one. I looked up a couple things online about trying to emulate the actual click event, but with no better results. I could do a REST service on the page, but I don't want to have to roll out the Extension Library to everyone in the company. Maybe I'll look into sending along the data in a XSP.partialRefreshPost call and eliminate the button entirely. We shall see.

Trying To Escape From Designer

Nov 8, 2011, 11:14 AM

Though I've grown to more or less enjoy writing Domino applications, I always feel like this is in spite of the tools, namely Designer. In a lot of ways, Designer has improved significantly over the last couple versions: as long as you ignore the speed, the Eclipse-ified Java and LotusScript editors are miles ahead of the antiquated previous ones, and it's handy to be able to switch to the Java perspective. However, so much else makes it a drag:

  • It's a Windows app. I use a Mac, so there's simply a big hurdle to using Designer. Parallels has smoothed the process greatly - running Notes in Coherence mode means I can use my preferred OS while still getting work done. However, it means that it's a big to-do whenever I want to make any tiny change in the code. For day-to-day work stuff, I can get a lot done in the Mac Notes client, since I put in a lot of work to make managing client web sites doable without going to the design side, and the Mac client still lets you edit views and agents. However, there's no XPages editor, so I can't use that for serious work.
  • It has a mind of its own. For some reason, a new clean install of Designer I made the other day got it into its head that opening any database should involve recompiling every XPage and Java class. Sometimes, it turns off "Build Automatically" for no reason, and then turning that back on also requires a full recompilation. Sometimes, it just holds up all user actions while it does... something for five minutes. What's it doing? Beats me.
  • How many times have I seen this window? A billion?
    Removing a database from the project list has about a 30% chance of causing a crash and quitting Designer has about a 50% chance. Sometimes, saving a form will do it. Sometimes, walking away from the computer to get a drink is enough. And every time it crashes, I have to dismiss the dialogs and check the task manager to get rid of any residual processes, such as an instance of nsd pegging a processor, then relaunch Notes and get back to the environment I had, which is not a speedy process.
  • I'm not that crazy about Eclipse. DDE is better than previous Designers, yes, but Eclipse is still a giant beast with the same sense of style and simplicity as the monstrous Java language that spawned it. On the plus side, the blue look that IBM came up with is actually rather attractive compared to the standard Eclipse UI, but little quality-of-life things are a drag. For example, how do you specify how many spaces a tab should take up visually? It's in a couple places in the preferences and you have to set them all, including buried inside a weird sub-preferences dialog for messing with your Java formatter. And how about changing your code syntax coloring? You have to go to a dozen places, one for each syntax type. Compare to a text editor like TextMate, where you can swap between packaged color schemes with a drop-down.

That's enough for the rant. So what is there to do about it? I give this problem a thought from time to time, and I don't think there's really a great option, but there are some places to start. The real key to any alternative scheme is DXL - using that, you can (more or less) view and modify design elements freely. I've toyed with this notion before - maybe a web UI that lets me pick the DB and design element I want so I can tweak the DXL manually for when I want to make a quick change but don't already have Windows or Designer open. It would mostly work, but it would take a lot of work to make it practical.

There's a big sticking point, too: XPages. If you export an XPage to DXL, you can see that the exporter basically punts on it - it's exported as a generic "note" type with Base64-encoded binary fields. I haven't run the field data through a decoder, so maybe one of them is the XML "source" of the page, and maybe it could be made to work, but that just raises more questions. Would that require updating the other binary fields in some way? Would importing it with a DXL importer cause it to generate and compile the Java representations? What about generic Java classes? Would those work with DXL and would they be compiled?

I'm starting to get an idea of what would be ideal and almost practical, though. One could write a WebDAV server (possibly with a complex servlet, a small web server to the side, or other trickery) that represents the design elements as editable files in a folder structure similar to the Java perspective view of the database. Traditional files and image resources could be edited as-is (which I think the built-in WebDAV server does), but design elements could be represented just as DXL and then re-imported when modified. Even if it doesn't support XPages, such a scheme might have a lot of promise and wouldn't be reliant on Designer or any other IDE.

If I ever get brave enough to delve into WebDAV or frustrated enough with Designer, I might just look into it myself.

So Here's Why I Hate LotusScript

Oct 29, 2011, 1:44 PM

Tags: domino

For the most part, writing agents in LotusScript is the best way to go (at least when it can't be done in formula language), mostly for smoothness of interaction with the built-in libraries (no .recycle()) and because it's less prone to running into memory problems when other agents go wonky than Java agents are. That doesn't mean I have to like it, though.

If there's one thing that drives me nuts about LotusScript more than any other aspect, it's its handling of arrays. This came to the fore with one of my recent projects, which involves spitting out the contents of a view, which in turn entails lots of use of entry.ColumnValues. My first, quick-and-dirty draft ran into an early performance issue, which is that every call of .ColumnValues on a NotesViewEntry seems to be as expensive as the first, meaning that the class doesn't do any internal cacheing and has to re-fetch it every time. Ugh, fine - I'll just assign the value to a new variable at the start:

Dim colValues as Variant colValues = entry.ColumnValues

Not too bad - two extra lines at the start of a loop is a small price to pay for a significant speed improvement. Unfortunately, it doesn't work - it throws a Type Mismatch error at runtime. After doing a TypeName() on entry.ColumnValues, I saw that it considers it "VARIANT( )", an array of Variants, which makes sense. It's a bit weird, since I've stored arrays in Variants before, but whatever - with a quick code adjustment, we're off to the races:

Dim colValues() as Variant colValues = entry.ColumnValues

Great! Now hit Ctrl-S and... compiler error. You're not allowed to assign to the entirety of an array like that. Argh! So I guess I'm going to have to make my new array manually and loop through the original to copy each entry over individually. If you're familiar with another scripting language, that probably sounds like a simple task, but LotusScript's array annoyances continue. Because this is like BASIC, you can't just do "colValues.add(something)" - you have to ReDim the array to the right size. Here's the code I ended up with:

Dim columnValues() As Variant ReDim columnValues(-1 To -1) ForAll columnValue In entry.ColumnValues If UBound(columnValues) = -1 Then ReDim columnValues(0 To 0) Else ReDim Preserve columnValues(0 To UBound(columnValues)+1) End If columnValues(UBound(columnValues)) = columnValue End ForAll

Before you look at that "-1 to -1" crap and deem me insane, hear me out. Though the NotesViewEntry doesn't cache its property value, the ForAll loop does, meaning that, according to the profiler, ColumnValues is only called once for each view entry, which is about as efficient as it gets. All that extra crap about ReDim'ing the array over and over instead of just once is essentially "free" compared to the expense of the product object call, so it ends up being completely worth it.


Oct 25, 2011, 9:52 PM

So there's a new round of talk lately about syncing and the trouble involved, thanks to some changes in Google Reader's behavior and the desire to find a new safe haven for RSS syncing. The best example is, unsurprisingly, from Brent Simmons:

Google Reader and Mac/iOS RSS readers that sync

However, the whole time I was reading this article, my brain kept yelling at me, louder and louder as time passed:

This is Lotus Notes! The system you're describing is Lotus Notes! It does syncing and deletion stubs and read marks! IT'S LOTUS NOTES!

This kind of thing would indeed be really easy in Notes/Domino, particularly if you were actually using the Notes client (though it wouldn't be much to look at). Subsets of data, managing deleted elements, timed refreshes from the source, storing each feed entry as its own entity, and offline access that can have its changed synced back to the master are all things that Notes has done since its conception - the only problem is that it's so ugly and arcane that mass-market appeal is nigh-impossible.

Nonetheless, it got me thinking about the viability of using Domino as a syncing server for this. You wouldn't be able to use NSF files in your RSS clients, which would make the job a bit tougher, but the new "XWork" licensing model would fit into this nicely. Scalability would be a serious concern, but the simple nature of the data would keep view updates quick, and it'd just be a bit of cleverness in the database layout to direct users to the correct place. Toss a couple clustered servers in there and you should have some good load balancing, too. The Domino Data Services API might be enough to handle data access from the client, but, if it's not, a couple simple agents would do it.

I'm sort of tempted to try hashing something out.

The Domino Data Service

Oct 5, 2011, 10:22 AM

Tags: domino

Though I don't have a use for it currently, I can't help but get kind of excited about the Domino Data Services in 8.5.3 and the Extension Library. If you're writing a normal Domino application - using either legacy elements or XPages - you probably won't have terribly much use for it.

However, the really cool aspect of it is that it significantly smooths the process of using Domino as a backing data store for another front end written in PHP, Ruby, or anything else. This has always been sort of possible - you could use the Java API or a combination of ?ReadViewEntries, ?CreateDocument, and ?SaveDocument URL commands to access Domino data without actually being in Domino, but it wasn't exactly a smooth process. With the Data Services, now Domino is very similar to, say, CouchDB, but with reader fields and impenetrable licensing terms for non-vendors.

One nice little side effect of the fact that it uses the HTTP stack is that DSAPI modules work. When I was testing around, I was able to get a list of available forums as Anonymous, resulting in only the two visible ones. When I included my user authentication filter cookie, it started showing me the rest of the forums that the user could access, exactly like you'd want. While you could presumably just pass the username and password in each request using normal HTTP authentication, it's pretty cool that any alternate methods like this work as well.

It's all pretty exciting, and I'm itching to find a use for it.

My next two favorite features of 8.5.3

Oct 5, 2011, 8:07 AM

Tags: domino

Since 8.5.3 has been out for about 24 hours now, I naturally rolled it out on both my development and production servers. Fortunately, my irresponsibility was greatly rewarded: the largest problems I've had so far were a change in the way Java classes are accessed in JavaScript (I could no longer just call methods on non-public classes defined in the same file as a public one, so I had to split them out into their own files... which is what you're supposed to do anyway) and a minor CSS change where the top borders of my Dojo tabbed tables are now back to a light grey color, so I need to find the new CSS rule to change them back to brown.

I'm rather happy so far about two minor things in particular: CSS/JavaScript aggregation and OSGi auto-loading.

The CSS/JavaScript aggregation is almost a freebie: once you have Designer 8.5.3, you get a new option in the database properties sheet to turn this on and then 8.5.3 servers will happily obey it. I immediately noticed a decent load-speed increase of about 1/3 and one non-technical guild member said that the odd problem of dog-slowness that they (and not other people) had has been fixed. My favorite aspect of this is that it's a smart feature: due to the way you define Dojo modules in an XPage as <xp:dojoModule/> elements and not just text on a page like normal HTML, Domino knows ahead of time what you're using and can thus feel free to optimize it in transparent ways like this. It feels good seeing the same code go from one form to a more efficient one just by virtue of done the "right thing" when writing it to begin with.

The OSGi plugin auto-loading was mentioned briefly on Dec's Dom Blog back in June and I hadn't seen much reference to it since, so I was afraid it wouldn't necessarily make it in. Fortunately, it has: I created a new Update Site with the template from an 8.5.3 server, imported the latest Extension Library, ran the "Sign All" agent, and set the notes.ini parameter to look there. And lo and behold: it properly loads up the extensions from the NSF, so I was finally able to delete the filesystem versions that were previously necessary. This makes managing the Extension Library much smoother and it's one less potential gotcha when I upgrade my dev server first and then want to deploy it - since the Update Site has a replica on both servers, the upgrade is handled with the normal replication process and I don't have to remember to copy any files over from server to server. And the fact that it's an NSF theoretically gives you all kinds of other, more complicated deployment options, like server-based Reader field control or partial replication to control which servers see which plugins if you're so inclined. Very cool - I approve, IBM.

My Favorite Minor Feature in 8.5.3

Sep 28, 2011, 6:23 PM

Tags: domino

I don't have access to the beta versions of new Notes/Domino versions, so I haven't been able to tinker around with all the cool new things that are slated to appear in 8.5.3, but that hasn't stopped me from getting pretty excited about some of them. The big-ticket items are clear: the new Domino Data Services and relational database access (through the Extension Library, which may as well be standard) could make practical very different ways of using Domino either as a standalone data source with a different front end or as a standalone front end with a relational data source. While both types of setups are theoretically possible now, they're such a hassle that they're not worth it - but, with 8.5.3, they're almost top-tier choices for system architectures.

However, since I don't plan to rewrite my entire architecture, those new features probably won't affect my day-to-day life for a while yet. What has me most excited in a practical sense is much more lowly: being able to do a full-text search with sorted results. I've found that one of the big bottlenecks in my guild-forums app is the sheer size of the views, particularly the Posts one. I used to stuff pretty much all of the summary data into the view, but then I found that removing the non-sorted columns sped up responsiveness dramatically. That whetted my appetite for clearing out unneeded sorted columns - since each sorted column contains a full view index, having even a handful can increase the total index size dramatically. Since it appears that FTSearch's performance is almost (but not quite) as good as getting all entries by a key, I'll be able to remove the rarely-used sorted columns, speeding up all the common operations in exchange for a very minor hit in the rare case. Plus, it'll just feel good to put Domino's searching capabilities to proper use.

Getting Domino LDAP to Work for Authentication

Aug 25, 2011, 4:23 PM

Tags: domino

Recently, I've been toying with the idea of setting up a couple extra services on my guild's Domino server - voice chat, non-Sametime chat, what have you - and I figured I should give a shot to LDAP authentication with the Domino directory for these. However, this is something I've never done, and the documentation is a little rough - most LDAP info on the web refers to non-Domino servers, while most Domino-specific information was written in about 1996.

I'll leave out the depressing details of the various things I tried in my quest to get LDAP working as an authentication mechanism for my Linux server (as a relatively simple test case) and point you instead to this dead-but-still-archived page: The key information on that page is the list of fields that you have to add to your user documents to use them for this purpose. During my harried testing, all /var/log/auth.log was telling me was "Invalid credentials", but what it really meant was that the user account it found didn't have the right attributes. Thanks, Linux!

Freaking Reader Fields

Jul 14, 2011, 3:02 PM

Tags: domino

Periodically (read: every day), I wonder about switching from Domino to a SQL server for data storage on my guild web site. The primary reason for this is speed: I'm doing primarily relational things, so I've had to wrangle Domino quite a bit to do this with any amount of speed and code cleanliness. Additionally, while most of my documents are entirely distinct from each other, I had to make concessions here and there, such as storing the latest Post date in Topic documents so I can sort them that way, and, each time I have to do that, there's another little bit of code maintenance and clustering-unsafety.

However, my ideas always come to a screeching halt when I remember Reader fields. They're simply too good, and the replacements I've found on the open source SQL databases have been, to put it kindly, lacking in comparison. They generally involve having some access level field or, best case, a multi-value field of names that are allowed to see the document, and then making sure that all of your queries or views honor that. Every method has some severe downside, ranging from inflexibility (access level) to nightmarish piles of code everywhere (multi-value name/group/role fields). Everywhere I accessed the database, I'd have to worry about security and document access, bloating up the code and just asking for data-leak bugs.

Domino, for all of its faults, makes this something you just don't have to worry about. If you have a Reader field, you can toss names, groups, and roles in there with impunity, and the server will handle the rest like you'd want. You don't have to do your own directory lookups, security checks, or nested queries. If the current user isn't on the list, the document may as well not exist. Even if the user had the UNID of the document and designer access to the database, it'd be beyond their reach. This is enormously comforting. And even though it's just a guild web site and not a giant corporate database, I'd still rather deal with a bit of tricky code for performance than the headaches and drama involved with people seeing what they're not allowed to see.

So, until I either get entirely fed up with Designer or I find an equivalent to Reader fields in a free SQL server, I'll be sticking with Domino.

Starting work on a DSAPI filter for Domino 8.5.2 on 64-bit Ubuntu

Apr 14, 2011, 4:33 PM

Tags: domino

For my main project, which contains a forum, one of the problems I ran into was Domino's session handling. Namely, it's designed such that HTTP user sessions last for the duration of the browser session and time out after 30 minutes. That's fine for, say, a corporate app, where you don't want a client logged in indefinitely. However, having to re-log-in to a forum every time you visit it would be a hassle.

The session timeout is easy to fix - you can just up the timeout period in the web site config. The cookie took a little more trickery, but wasn't too bad either. I set up some code in the beforePageLoad event to look for a DomAuthSessId cookie and, if present, create a persistent cookie. The server can't tell the difference between the session and persistent cookies, so this works.

However, it's sort of ugly. While I absolutely don't want to have people re-log-in every time they restart their browser, I also don't really want tons of session-scoped managed beans floating around for months, and this kind of trick defeats any use of the Internet Users list in Domino Administrator (a minor quibble, but a quibble nonetheless). Ideally, I'd be able to automatically log people in on subsequent visits, but not have to actually maintain server sessions for them. From all I've read, this calls for a DSAPI filter. Since that involves C/C++, there's bound to be a lot of setup and tribulation.

First off, I had to grab the Notes/Domino C API toolkit. Fortunately, Google led me directly to the IBM download page and, after re-confirming that I don't want them to send me email, I was given a download link for a quaintly compress'ed file and some installation instructions. I dropped the libraries into /opt/ibm/lotus/notesapi and set up a symlink at /opt/lotus/notesapi just in case. The samples in the toolkit do indeed include a DSAPI filter, and I found another one doing something very much like what I want here:

To keep things simple, I started with that code and commented/deleted everything that actually does something - I just wanted the simplest possible filter to see if it works at all. All it does is announce that it's alive and then proceed to not handle any requests:


Now came the issues. First off, the DSAPI example in the toolkit only came with a handful of Makefiles, and none for Linux. The Solaris one was close, so I borrowed and modified that one with some tips from . The most vital thing was that, since I'm running in a 64-bit environment but using a 32-bit Domino server, I had to go out of my way to compile a 32-bit library - this is done by adding "-m32" to the CCOPTS and (maybe required) LIBS lines. The Makefile I ended up using is:


Along the lines of the "-m32" flag, I had to install some additional development libraries to support 32-bit compiling. In my travails, I ended up installing all sorts of apt packages before it worked, so it's possible that only the last (or another, non-C++ library) is necessary:


Unfortunately, even when you get the library building properly, it's tough to get it to load into Domino, and it's not very helpful as to why - all it says is "Failed to load DSAPI filter: " and then the file name... and it gives the same error if you point it to a non-existent file, so there's no way of knowing WHY it failed to load it. It's all trial and error (unless you actually know what you're doing, presumably).

Once all that was in line, though, I could build the library with make, copy it to /opt/ibm/lotus/notes/latest/linux, add "" to the Configuration tab of my Web Site document in the Directory, and restart HTTP. When HTTP is loading, it will print out "HTTP Server: DSAPI DSAPIDLL Loaded successfully" (or whatever message you put in there), so you know it's working.

Now comes the task of actually writing the filter. Considering that I haven't written any C or C++ since college, I expect lots of fun with buffer overflows and mysterious crashes. Exciting!

Installing Domino 8.5.2FP2 on an Ubuntu 10.10 server

Apr 5, 2011, 7:19 PM

Tags: domino

Because I am insane, I decided to install Domino on my newly-minted 768 slice from Slicehost. That's just above the 512MB RAM minimum that Domino demands and half the recommended 1.5GB, so we'll see how it goes.

The Linux installation I went with for my slice was Ubuntu 10.10 64-bit, which firmly puts it into the realm of unsupported, as far as Domino is concerned. Fortunately, three pages I found made the process relatively painless:


The first wrinkle you run across is that Domino adores Java, but stock Linux installs do not. Fortunately, link #3 there explains how to cleanly install Java on your server. Namely, add this line to /etc/apt/sources.list:

deb maverick partner

Once you do that, you can run apt-get install sun-java6-jre, wince at the huge number of dependencies it requires, and let it do its thing.

After that, you're almost ready to start installing. If you're running on a 64-bit installation, you'll need the standard 32-bit libraries, since I don't think there's yet a 64-bit Domino server for any-old-Linux. That one's easy, though: apt-get install ia32-libs and you're all set.

The Domino installation proper was relatively straightforward. I created a "notes" user and associated group beforehand, picked and created some directories for the program files and data owned by that user, and went through its command-line wizard via ./install -console . For the final step, I told it to expect a remote installation and fired up the Remote Server Setup tool on Windows, which worked flawlessly.

The next step was the FP2 installer. As is the case with the Windows version, the installer for the fix packs is totally different than the normal installer for some reason. It initially complained about the size of my Terminal window and, rather than trying to convince it that my GUI terminal was plenty good enough for its installer, I took its advice and did export LOTUS_NOROWCOLCHECK=1 . After that, it complained about needing to know where the data directory was, but provided a similar instruction for setting that variable. Once that was in order, the installation went smoothly.

The next tough part was how to get Domino to run at startup. I've been familiar from time to time with various *nixes and their service setups, but it seems to change every couple of years, so I wasn't sure where to go with this. Fortunately, link #2 above did all the work for me. Since I found it long after installation, I didn't do the "customized distribution" bit, but I DID grab the Domino init script. Once I put that in place, modified the variables to point to my directories and user, and followed the other instructions, I was all set. Now I can control Domino via "service domino (start|stop|restart)" like you'd expect.

Due to the nature of Linux, the question of where to keep the program and data directories is a weird one. The defaults are something like "/opt/ibm/lotus" and "/local/notesdata", respectively, which would certainly WORK, but don't fit in with much else. I've fiddled with my placement a bit and settled on the same thing for the former, but "/var/lib/domino/data" for the latter. That allowed me to set up "/var/lib/domino/daos" and "/var/lib/domino/logdir" (for transaction logging) and keep things relatively clean. One thing to note: if you switch around the directories after the fact, make sure to change both the init script and the notes.ini in your program directory to reflect the new locations.

All told, it seems to be working pretty well. When you're using it like a normal Notes server, there's not much distinction other than the lack of OLE. It has all the same bugs (like the aggravating XPages Java classloader "X is not compatible with X" errors I run into constantly) and same features, which is really the idea.

Pretty URLs

Mar 21, 2011, 8:09 PM

Tags: domino

Way back when I wrote my own blog back-end, I went out of my way to make decent-looking URLs that didn't betray the use of PHP for the code. It's not that using PHP was inherently bad, but the ".php" in the URLs was ugly and would have made it tough to move to any other back-ends. It's just one of those good-idea cleanliness things. Back then, I did it with Apache MultiViews and just had names like "archive.php", so I could make URLs like "/archive/2002/12" for that month's posts.

Since I've started using Domino, however, it's gotten a bit tougher. The parts of the URL within a Notes database can actually be pretty great - something like "/People/Foo+Fooson" is about as good as you'd want. However, it's the database path that kills me. You can set up URL substitution rules, but that's a weird combination of setting up a Directory Web Rule plus making sure that every place you make a URL in your code uses that name. For something like a view column, that'd mean either hard-coding your server-specific setting or sucking it up and using @WebDbName. For the most part, I've given up and gone with the latter, for the sake of portability.

XPages bring good news and bad news.

The good news: your application almost never needs to care about the database file path, ".nsf" extensions, or anything of the like. There may be edge cases where you do, but for the most part you can just start your URLs with "/" and the database path is filled in for you (at the cost of starting non-DB paths with "/.ibmxspres/domino").

The bad news: even though the database path reference is now handled by the runtime and not the programmer's code, I don't know of a way to override the behavior. I'm sure it's there SOMEWHERE, buried in the hordes of Java classes and method calls spawned for each page request, but I don't see it. What I'd really want is to tell the XPages runtime that, while "/wow/forums.nsf" is the "right" way to reference the database, I'd rather it just start with something like "/forums". It seems like something that must be handled in the file, but I haven't found anything about it yet. Admittedly, it's tough to search for - most pages with the keywords I want are talking about @WebDbName replacements in SSJS and how to references external Domino resources. If it's not present already, maybe it's something they'll add in a future point release. One can hope.


Feb 13, 2009, 12:40 AM

Tags: domino

My company is currently dealing with a client that wants us to look into Single-Sign-On options to integrate with their portal. Since this client has a lot of money, I've been doing just that lately. The keywords are straightforward: they want single-sign-on using SAML. Gotcha!

Step one: search for the keywords. There are a couple promising links:

The first one is an overview article that has two followups, but the only one that mentioned SAML is the first, and only in passing. Right. The second one is a forum posting asking if it's possible to use SAML to authenticate with a Domino server. No replies.

My further searching led me to Tivoli Federated Identity Manager. I've known of Tivoli primarily as that "other" group of files when I go to download something on IBM's site, part of the sea of giant "IBM Application Services for Application Server Version 6.1, for IBM Application Services 6.1 Enterprise for e-business for Windows XP, Windows 2003 Multilingual (1 of 3)" names.

But the page I found, part of a help DB for the program, contained a kernel of promise: Domino is listed as one of the usable directory stores. Now, I'm not sure if that means this thing will allow you to sign INTO Domino, or if it will just let you sign into OTHER stuff using names FROM Domino, but it's worth investigating, right?

So skip ahead a while and I'm looking at an installer for Tivoli Access Manager. Well, I THINK I'm looking at an installer. I'm also looking at a bunch of other potential installers:


So I take a stab at it. "install_ammgr.exe" must stand for "Install Access", uh, "Mmanager", right? Double-click and... nothing. I try some of the others. Double-click and... nothing.

So I put it aside for now. I find some other downloads, such as one that seems to be what I was originally looking for, the Federated Identity Manager. I had a horrifying time with this, including a foray into WebSphere, before I decided to come back and take a whack at Access Manager again. I found the problem this time! You see, since IBM decided that useful file names are for wusses, their downloads are named things like "C1AV9ML", and I, as I always do with Domino, renamed it something useful, "Tivoli Access Manager Base - C1AW2ML", and extracted it into a folder of the same name.

If you're familiar with IBM, or if you have arrived at the present day via a time machine from circa 1990, you may see the problem immediately: the spaces. I mean, sure, Windows has had spaces-in-the-filename support since at least 1995, so not accounting for them was an amateur mistake fourteen years ago, but here we are. I renamed the folder, clearing out spaces, and voila! The installer launched!

It was very promising, too, asking me what directory I wanted, what the Domino server name was, and everything. But when I installed it, it failed, telling me only that it failed and that I should probably check a log file (which I guess it was unable to do itself). When I did, I noticed this suspicious text:

HPDHZ0021E   This file could not be found:  M:\Desktop\IBM\SAML\TAM-Base-C1AV9ML\windows\TivSecUtl\Disk

Oh god. What is in the "TivSecUtl" folder, exactly? Surprise surprise:

Disk Images

Excuse me? There's a folder in the directory tree that was in the zip file that IBM made that it simply can't use because they don't know how to escape spaces? Let me reiterate:

IBM's installer can't work out of the box.

There's already the whole issue that they use these horrible Java-based installers for their "Enterprise" apps. When the main problem is how weird it is that Lotus Notes uses a platform-native installer while Lotus Domino uses something that looks like it was mugged by X11, it's just weird. But when the JVM initialization takes up as much time as the entire rest of the installation, and you have to sit through it half a dozen times because you keep having to make more changes to the installer script, it's like IBM is punching you in the face.

The only theory I can come up with is that IBM doesn't actually want you to use their software. What other explanation is there for an installer package that simply doesn't work? It's not a OS version issue, or a weirdo setup issue, it just doesn't work.

This is even beneath the callous explanation that they do it so they can charge for support contracts. The RIGHT way to lock customers in to painful support is to at least make the program INSTALL and provide a basic level of functionality that is ALMOST what people want, but then make getting from there to the goal like wading through an ocean of steel briars. Like Notes, for example. At least that shows a basic level of respect, not just letting the programmer zip up whatever in-development folder he had and uploading it to the software catalog.


It makes all the stupid crap like mysterious line breaks in rich text on the web seem almost user-friendly.