Showing posts for tag "liberty"

Writing Domino JEE Apps Outside Domino

Nov 8, 2019 1:47 PM

In my last post, I mentioned that I decided to write the File Store app as a Jakarta EE application running in a web server alongside Domino, instead of as an app running on the server itself. In the comments, Fredrik Norling asked the natural question of whether it'd have been difficult to run this on the server, which in turn implies a lot of questions about deployment, toolkits, and other aspects.

Why Not

My immediate answer is that it would certainly be doable to run this on Domino, but the targeted nature of the app meant that I had some leeway in how I structured it. There are a couple reasons why I went this route, but most of them just boil down to not having to deal with all the gotchas, big and small, of doing development on top of Domino.

As a minor example, I wanted to add some configuration parameters to the app, and for this I used MicroProfile Config. MicroProfile Config is a small spec that standardizes the process of doing key-value-based configuration, allowing me to just say that I want a named value and let the runtime pick it up from the environment, system properties, or a config file as necessary. It wouldn't be difficult to write a configuration checker, but why bother reinventing the wheel when it's right there?

Same goes for having the SFTP server load at startup and be running consistently. If I did this on Domino, I could either put it in an NSF-based XPages app with an ApplicationListener and depend on the preload notes.ini parameter, or I could go my usual route and use an HttpService that manages the lifecycle. Neither route is difficult either, but they're both weirder and more fiddly than using servlet context listeners in a normal JEE app.

And then there's just the death by a thousand cuts: needing to use Tycho to build, having to deal with Eclipse Target Platforms, making sure anything using reflection is wrapped in an AccessController block to avoid Java policy issues, the nightmare of dependency management, the requirement to use either Designer or the okay-but-still-cumbersome Domino HTTP restart development cycle, and on and on. With a JEE app, all of those problems disappear into thin air.

The Main Hurdle

All that said, there's still the hurdle of actually implementing Notes native API access outside of Domino. At its core, what I'm doing is the same as what has been possible for years and years, initializing a Notes/Domino runtime in a secondary process. The setup for this varies platform by platform but generally involves either setting a couple environment variables for your process or (as is the case with the Domino Open Liberty Runtime) spawning the process from Domino itself.

Beyond setting up your process's environment, there's also the matter of initializing each thread of your app. On Domino, all threads are inherently NotesThreads, but outside of that there's some specific management to be done. You can call NotesThread.sinitThread() and NotesThread.stermThread() manually or run your code in specifically-spawned NotesThreads. I largely do the latter, making use of an ExecutorService to handle maintaining a thread pool for me. I then added some supporting code on top of that to let me run blocks of code as an arbitrary named user while retaining a cached set of sessions. That part wouldn't really be necessary if you're writing an app that doesn't need to act as different user names, but it's handy for something like this, where incoming connections must run as a user to enforce security fields properly.

Philosophical Advantages

Beyond my desire to avoid hassles and get to use modern Jakarta EE goodies, I think there are also some more philosophical advantages to writing applications this way, and specifically advantages that line up with some of HCL's stated long-term goals as well. Domino has long been a monolith, and that has largely served it well, but keeping everything from the DB all the way up through to the app-dev stack in the same bag means you're often constrained in your toolkits and deployment choices. By moving things just outside of the main Domino tower, you're freed up to use different languages and techniques that don't have to be integrated and maintained in the core. This could be a much larger jump than I'm doing here, and that's just what HCL has been pushing with the AppDev Pack and the associated Node.js domino-db module.

I think it's beneficial to picture Domino more as a dominant central core with ancillary servers and apps running just adjacent to it - not a full-on Microservices architecture, but just a little decentralized to keep areas of concern nice and separate. Done well, this setup is a lot more flexible and fault-tolerant, while still being fairly straightforward and performant. It's also a perfect match for a project like this that's geared towards implementing a new protocol - it doesn't even have to worry about HTTP SSO or reverse proxies yet.

So I think that this is where things are heading anyway, and it's just a nice cherry on top that it also happens to be a much (much, much) more pleasant way to write Java apps than OSGi plugins.

Another Side Project: NSF SFTP File Store

Nov 5, 2019 4:12 PM

When I Know Some Guys kicked off, we bought a couple of Transporter devices to handle our file-syncing needs without having to rely on Dropbox or another hosted service. Unfortunately, Nexsan killed off Transporters a couple years back, and, though they still kind of work, it's been a back-burnered project for us to find a good replacement.

Ideally, we'd find something that would handle syncing data from our various locations transparently while also allowing for normal file access through some common protocol. Aside from the various hosted commercial services, there are various software packages you can run locally, like OwnCloud and NextCloud. I even got a Raspberry Pi with a USB hard drive to tinker with those, though I never got around to actually doing so.

Yesterday, though, I realized that we already have a fleet of privately-owned servers that replicate seamlessly in the form of our Domino domain. They also, conveniently, have nice capabilities for blob storage, shared user authentication, and fine-grained access control. What they didn't have, though, was any good form of file protocol. I'm pretty sure that Domino still has WebDAV built-in, but that's just for design elements. Years ago, Stephan Wissel created a project that works with file attachments, but that didn't cover all the bases I wanted and I didn't want to adopt the code base to extend it myself. There's also Karsten Lehmann's Mindoo FTP Server from around the same time, but that was non-SSH FTP and targeted at the local filesystem.

So that meant it was time for a new project!

The Plan

I initially looked at WebDAV, since it's commonly supported, but it's also very long in the tooth, and that has led to all of the projects implementing that being pretty old and cumbersome as well.

Then, I found the Apache Mina project, which implements a number of server protocols, SSH included, and is actively maintained. Looking into how its SFTP support works, I found that it's shockingly simple and well-designed. All of the filesystem access is based on the Java NIO packages added in Java 7, which is a pluggable system for making arbitrary filesystems.

Using SFTP and SCP means that it'll work with common tools like Transmit and - critically - rsync. That means that, even in the absence of an custom app like Dropbox, mobile access and syncing with a local filesystem come "for free".

The Project

So out of this was born a new project, NSF File Server. Thanks to how good Mina is, I was able to get a NIO filesystem implementation and SFTP+SCP server up and running in very little time:

Screen shot of the SFTP server in Transmit

In its current form, there aren't a lot of tricks: the files are stored as attachments to normal documents in a "filestore.nsf" database with two views, which allow for directory-contents and individual-file lookup while also being pretty self-explanatory to a Notes client. I have some ideas about other ways to structure this, but there's an advantage to having it be pretty basic:

Screen shot of the File Store NSF

Similarly simple are the authentication mechanisms, which allow for both password and public-key authentication based on the HTTPPassword and sshPublicKey fields in a person document, respectively (and maybe via LDAP in directory assistance? I never remember the mechanics of @NameLookup).

The App

Because this is a scratch-our-itch project and I'm personally tired of dealing with Domino's OSGi environment, the app itself is implemented as a WAR file, expected to be deployed to a modern Jakarta EE server like my precious Open Liberty. Conveniently, I have just the project for that as well, making deploying NSF-accessing WARs to Domino a bit more reasonable.

For now, the app is faceless: the only "web app" bits are some listeners that initialize the Notes environment and then spawn the SSH server. I plan to add at least a basic web UI, though.

Future Plans

My immediate plan is to kick the tires on this enough to get it to a point where it can serve as its original goal of a syncing SFTP server. I do have other potential ideas in mind for the future, though, if I feel so inspired. Most of my current logged issues are optional enhancements like POSIX attribute support, more-efficient handling with the C API, and better security handling.

It's also a good foundation for any number of other interfaces. A normal web UI is the natural next step, but it could easily provide, for example, S3 API compatibility.

For now, though I haven't gotten around to uploading a build to OpenNTF yet, feel free to poke around the code and let me know if any ideas strike your fancy.

New Project: Domino Open Liberty Runtime

Jan 3, 2019 5:23 PM

Tags: liberty

The end of the year is often a good time to catch up on some side projects, and this past couple weeks saw me back to focusing on what to do about our collective unfortunate situation. I started by expanding the org.openntf.xsp.jakartaee project to include several additional JEE standards, but then my efforts took a bit of a turn.

Specifically, I thought about Sven Hasselbach's series on dropping Domino's HTTP stack while still keeping API access to Domino data, and decided to take a slightly-different approach. For one, instead of the plucky-but-not-feature-rich Jetty, my eye turned to Open Liberty, the open-source variant of WebSphere Liberty, which in turn is the surprisingly-pleasant trimmed-down counterpart to WebSphere. Using Liberty instead of Jetty means getting a top-tier Java EE runtime, supporting the full Java EE 8 and MicroProfile 2.1 specs, developed by a team chomping at the bit to support all the latest goodies.

Additionally, I decided to try launching Liberty from a Domino plugin, and this bore fruit immediately: with this association, the Liberty runtime is able to fire up sessions and access databases as the Domino server without causing the panic halt that Sven ran into.

So, in short, what this project does is add a fully-capable Java EE server with all the fixings - the latest JEE spec, HTTP/2, Servlet 4, WebSockets, and so forth - running with native access to Domino data alongside a normal server, and with the ability to manage configuration and app deployment via NSFs. Essentially, it's like a second HTTP stack.

Why?

I made some good progress in bringing individual JEE technologies to XPages, but I was still constrained by the core capabilities of the XPages runtime, not the least of which was its use of Servlet 2.4, a standard that went obsolete in two-thousand-freaking-five. Every step of the project involves fighting against the whole underlying stack, just to get some niceties that come for free if you start with a modern web container.

Additionally, while Domino has the ability to run Java web applications, this support is similarly limited, providing very few of the standards that make up Java EE and even apparently lacking a JSP compiler set up on the server. It's also, by virtue of necessarily wrapping the app in an OSGi bundle, much fiddlier to develop than a normal WAR file.

And, in a general sense, I'm tired of waiting for this stack to get better. Maybe HCL has grand plans for Java development on Domino in the future - they haven't said. I still doubt it, in part because of the huge amount of work it would entail and in part because I'm not sure that improving XPages would even be strategically wise for them. And say they did improve XPages in a lot of ways people have been clamoring for - WebSockets and whatnot. Would they cover all of the desired features? What about newly-emerging technologies from outside? Their Node.JS strategy makes me think they've thought better of being the vendor of a full-stack web technology.

This route, though, provides a route to making web apps with current standards regardless of what HCL does with XPages. This way, you can work with the entire Java web community at your back, rather than cloistered off with unknown technology. If you want to make an app with Spring, you can, following all of their examples. If you'd rather use PrimeFaces, or just JAX-RS, or JSP, you can do so just as easily. And if your chosen technologies go out of favor, you'll be in the same boat as countless others, and the new preferred choices will be open to you.

Finally, there's just the fact that Java EE 8 is really, really good. The platform made tremendous strides since the bad old days, and developing an app with it is a revitalizing experience.

How?

To set this up, I deliberately chose a very low-integration path: the task in Domino unzips a normal Open Liberty distribution and then runs it using Domino's JVM, just using the default bin/server script. No embedding, no shared runtime. This way, it doesn't have to fight against any constraints that Domino's environment imposes (such as the fact that both Domino and Liberty want to run an OSGi environment), and it doesn't lead to a situation where a crash in Liberty would bring down Domino's HTTP.

The rest kind of comes along for the ride. Since it's running with the Domino JVM, it already has the trappings needed to use Notes.jar, so it's really just a matter of using the classes and making sure you run inside a NotesThread or otherwise initialize and terminate your thread.

Future

Assuming I keep with this project (and I think I have some for-work uses for it, which dramatically increases its odds), I have some ideas for future improvements.

I've added a basic HTTP reverse proxy servlet, and I plan to make it more integrated. The idea there is to allow Liberty to be the primary HTTP entrypoint for Domino, with anything not handled by a web app it's hosting to pass through transparently to Domino.

In time, I aim to add some more integration, such as CrossWorlds and general utilities. I've started by adding in a basic user registry, allowing JEE-standard apps to authenticate against Domino without extra configuration (though it doesn't currently do groups). That could be expanded a good deal - Liberty could read SSO tokens using the C API (or share LTPA as WebSphere normally does), and it'd be nice to have a reasonable method for sharing non-SSO DomAuthSessId cookies.

The Project

I set up on the project on GitHub: https://github.com/OpenNTF/openliberty-domino . I think there's some definite promise with this, especially once there are a couple example apps that could show off the possibilities.