Cramming Rails Into A Maven Tree

  • Sep 26, 2016

Because I'm me, one of the paths I'm investigating for my long-term blog-reformation project is seeing if I can get Ruby on Rails in there. I've been carrying a torch for the language and framework for forever, and so it'd be good to actually write a real thing in it for once.

This has been proving to be a very interesting thing to try to do well. Fortunately, the basics of "run Rails in a Java server" have been well worked out: the JRuby variant of the language is top-notch and the adorably-named Warbler project will take a Rails app and turn it into a JEE-style WAR file or self-hosting JAR. That still leaves, though, a few big tasks, in order of ascending difficulty:

  1. Cramming a Warbled Rails app into a Maven build
  2. Getting the Rails app to see the Java resources from the other parts of the tree
  3. Initializing Darwino tooling alongside Rails
  4. Making this pleasant to work with

So far, I've managed to get at least a "first draft" answer to the first three tasks.

Cramming a Warbled Rails app into a Maven build

When you get to the task of trying to do something unusual in Maven, the ideal case is that there will be a nice Maven plugin that will just do the job for you. Along those lines, I found a few things, ranging from a tool that will assist with making sure your Gems (Ruby dependencies) are handled nicely to one that outright proxies Gems into Maven dependencies. However, none that I found quite did the job, and so I fell back to the ol'-reliable second option: just shell out to the command line. That's not ideal (for reasons I'll get to below), but it works.

I ended up putting the Rails app into src/main/rails/blog and then using the exec-maven-plugin to do the Warbling for me:

<plugin>
	<groupId>org.codehaus.mojo</groupId>
	<artifactId>exec-maven-plugin</artifactId>
	<executions>
		<execution>
			<id>create-final-war</id>
			<phase>package</phase>
			<goals>
				<goal>exec</goal>
			</goals>
			<configuration>
				<executable>/bin/sh</executable>
				<workingDirectory>.</workingDirectory>
				<arguments>
					<argument>-c</argument>
					<argument>
						rm -f src/main/ruby/blog/*.war
						cd src/main/ruby/blog &amp;&amp; \
						jruby -S bundle install &amp;&amp; \
						jruby -S warble executable war &amp;&amp; \
						cd ../../../.. &amp;&amp;
						mv src/main/ruby/blog/*.war target/${project.build.finalName}.war
					</argument>
				</arguments>
			</configuration>
		</execution>
	</executions>
</plugin>

This amounts to a shell script that clears out any previous build, makes sure the dependencies are up to date (jruby -S bundle install), creates a self-hosting WAR file (jruby -S warble executable war), and then copies that result to the name expected by normal Maven WAR packaging. This basically works.

Getting the Rails app to see the Java resources from the other parts of the tree

Now that I had a properly-building WAR, my next task was to bring in any dependency JARs and in-project Java classes for use at runtime. Fortunately, this is a job that Warbler can handle, by way of its config/warble.rb file. In the root of the blog project, I ran warble config to generate this stub file. Like almost everything else in Rails, the configuration is done in Ruby, and this file is a large block of Ruby code examples, mostly commented out. I adjusted the lines to copy in the dependency JARs (which Maven, in a WAR package, will have previously copied for me) and to copy in any "loose" Java classes I may have alongside Rails in the current project:

config.java_libs += FileList["../../../../target/frostillicus-blog/WEB-INF/lib/*.jar"]
config.java_classes = FileList["../../../../target/frostillicus-blog/WEB-INF/classes/**/*"]
config.pathmaps.java_classes << "%{../../../../target/frostillicus-blog/WEB-INF/classes/,}p"

These lines use a helper class named FileList to glob the appropriate files from the project's target directory and copy them in. In the case of the loose classes, I also had to figure out how to clean up the path names - otherwise, it created a bizarre directory structure within the WAR.

With those lines in place, Warbler set up everything nicely - I could reference code from any of the dependencies, the other modules, or anything from the src/main/java folder within the same module.

Initializing Darwino tooling alongside Rails

The last step I got working is related to the previous one, but has a couple wrinkles. In addition to just having the Darwino classes available on the classpath, a Darwino application has an initialization lifecycle, done in a JEE app via filters defined in web.xml. It may also have some support files for defining beans and properties, which aren't covered by the same process as above. To start with the latter, I needed to figure out how I was going to get the files included in the "normal" JEE project's WEB-INF folder copied into the Rails WAR without destroying anything else. Fortunately, the same config file had a hook for that:

config.webinf_files += FileList["../../webapp/WEB-INF/**/*"] - ["../../webapp/WEB-INF/web.xml"]
config.pathmaps.webinf_files = ["%{../../webapp/WEB-INF/,}p"]

This one is basically the same as above, but with an important subtraction: I want to make sure to not copy the normal app's web.xml file in. If that's copied in, then Warbler will respectfully leave it alone, which would mean that the Rails portion of the app won't be launched. I'm handling that specially, so I made sure to use Ruby's "array subtraction" shorthand to make sure it's not included.

So that left modifying the web.xml itself, in order to serve two masters. Both Darwino and Rails expect certain filters to happen, and so I copied Warbler's web.xml.erb template into the config directory for modification. .erb is the designation for "embedded Ruby", and it's a technique Ruby tools frequently use for sprinkling a bit of Ruby logic into non-Ruby files, with a result that's similar to PHP and other full-powers templating languages. The resultant file is essentially a mix of the stock file created by Darwino Studio and the Warbler one, with some of the Darwino additions commented out in favor of the Rails stack.

Making this pleasant to work with

This final part is going to be the crux of it. Right now, the development process is a little cumbersome: the Rails app is essentially its own little universe, only fused with the surrounding Java code by the packaging process. That means that, even if I got a great Rails IDE, it wouldn't necessarily know anything about the surrounding Java code (unless they're smarter than I'd think). More importantly, the change/view-live loop is lengthy, since I have to make a change in the app and then re-run the Maven build and re-launch the embedded server. I lose the advantages both of Eclipse's built-in run-on-Tomcat capabilities as well as the normal Rails self-hosting hot-code-replace capabilities.

Fortunately, at least for now, the awkwardness of this toolchain may be primarily related to my lack of knowledge. If I can find a way to automate the Warbling inside Eclipse, that would go a tremendous way to making the whole thing a mostly-smooth experience. One potential route to this would be to create a Maven plugin to handle the conversion, and then include an m2e adapter to get it to conform to Eclipse's expectations. That would be a tremendous boon: not only would it be smoother to launch, but it would potentially gain the benefit of referencing workspace projects directly, lessening the need to worry about Maven installation. That would be a good chunk of work, but it's an area I'd like to dive into more eventually anyway.

In the mean time, the latest state of the conversion is up on GitHub for anyone curious:

https://github.com/jesse-gallagher/frostillic.us-Blog

Quick Post: Maven-izing the XSP Repo

  • Sep 17, 2016

This post follows in my tradition of extremely-narrow-use-case guides, but perhaps this will come in handy in some situations nonetheless.

Specifically, a while back, I wrote a script that "Maven-izes" the XPages artifacts, as provided by IBM's Update Site for Build Management. This may seem a bit counter-intuitive at first, since the entire point of that download is to be able to compile using Maven, but there's a catch to it: the repository is still in Eclipse ("P2") format, which requires that you use Tycho in your project. That's fine enough in most cases - since Domino-targetted projects are generally purely OSGi, it makes sense to have the full OSGi stack that Tycho provides. However, in a case where Domino is only one of many supported platforms, the restrictions that Tycho imposes on your project can be burdensome.

So, for those uses, I write a JRuby script that reads through the P2 site as downloaded and extracted from OpenNTF and generates best-it-can Maven artifacts out of each plugin. It tries to maintain the plugin names, some metadata (vendor, version, etc.), and dependency hierarchy, and the results seem pretty reliable, at least for the purpose of getting a non-Tycho bundle with XSP references to compile. This isn't necessarily a route you'd want to take in all cases (since you don't get the benefits of normal OSGi resolution and services in your compilation), but may make sense sometimes. In any event, if it's helpful, here you go:

https://github.com/jesse-gallagher/Miscellany/blob/master/UpdateSiteConversion/convert.rb

Reforming the Blog in Darwino, Part 1

  • Sep 15, 2016

This continues to be a very interesting time for Domino developers, with the consternation of MWLUG giving way to IBM's recent announcement about their plans for Domino. Like everyone, I have my feelings about the matter, but the upshot is that moving-forward tone still stands. With that in mind, let's get down to business, shall we?

I'm going to kick off my long-term blog series of moving my blog itself over to a Darwino+JEE application. I say "long-term" because it's a pure side project, and I still have a lot of decisions yet to make with it - for one, I haven't even decided on which toolkit I'll be using for the UI. However, since this will have the additional effect of being a demo for Darwino, I want to lay the groundwork early. And hey, that's one of the benefits - since the data will keep replicating, I can take whatever time I need on the new form while the old one chugs along.*

As I have time to work on it, I'll go over the steps I've taken and put the current state up on GitHub. These descriptions are meant to be somewhere in between an overview and a tutorial: I'll cover the specific steps I took, but I'm leaving out a lot of background info for now (like installing a PostgreSQL database, for example).

To start out with, I'll create the basic projects. A prototypical Darwino application consists of a two-tiered structure of Maven modules: a root container module and then several related projects. In my case, the tree looks like this:

Since my plans are simple, this only has a few of the potential modules that could be created, but retains the separation-of-concerns structure. The purposes of these projects are, in order of importance:

frostillicus-blog-shared
This project holds the database definition and any business logic that should be shared across each UI for the application. Most of my Java code will go here (unless I pick a Java-heavy UI toolkit, I guess). Any model objects, servlet definitions, scheduled tasks, and so forth will go here.
frostillicus-blog-webui
This project holds the assets used by the web UI of the application, which could be shared by the JEE project and any "hybrid" web-based mobile UIs I make (which I won't for this). By default, this contains a skeletal Ionic-based UI.
frostillicus-blog-j2ee
This project holds the scaffolding for the JEE servlet app ("J2EE" still has a better ring to it than "JEE", though). Depending on where I go with the UI, this will either be a small shim just to get HTML served up or a larger server-side toolkit project.

In a different situation, there could be up to five additional modules: native and hybrid UIs for Android, the same pair for iOS, and an OSGi plugin shim for running on Domino. I may end up wanting to run this on my existing Domino server, but I don't have a need for offline mobile access just for my blog, so I'll definitely be skipping those. (Edit: I forgot one potential UI project: a SWT front-end for desktops)

For the most part, the default created classes do what I want them to do to get it started. There's really only one tweak to make: since I'll want to be able to search, I know up front that I want to enable FT searching in the Darwino DB. This sort of thing is done in the created AppDatabaseDef class. There's quite a bit that you can do there, but I'll mostly just uncomment the lines that enable full-text search in the loadDatabase method:

@Override
public _Database loadDatabase(String databaseName) throws JsonException {
	if(!StringUtil.equalsIgnoreCase(databaseName, DATABASE_NAME)) {
		return null;
	}
	_Database db = new _Database(DATABASE_NAME, "frostillic.us Blog", DATABASE_VERSION);

	db.setReplicationEnabled(true);
	
	db.setInstanceEnabled(false);
	
	{
		_Store _def = db.getStore(Database.STORE_DEFAULT);
		_def.setFtSearchEnabled(true);
		_FtSearch ft = (_FtSearch) _def.setFTSearch(new _FtSearch());
		ft.setFields("$"); //$NON-NLS-1$
	}

	return db;
}

That will ensure that, when the database is deployed (or, if I make changes later, upgraded), FT search is on. The line ft.setFields("$") uses a bit of JSONPath, basically saying "start at the root, and cover all fields". I'll probably come back to this class later to add some more optimizations, but that can wait until I'm sure how the structure of the app will take form.

The last step for now is to set up replication between Domino and this fledgling Darwino app. To do that, I'll set up an adapter definition in the Sync Admin database (the Domino-side application that manages Darwino replication):

The code on the page is the DSL used to define the mapping between Domino documents and Darwino's JSON docs. In this case, it's the code that is automatically generated by the "Generate From Database" tool, and everything after the first line isn't strictly necessary: without guidance, the replicator will try to translate the doc contents as best it can, and the data in this DB is pretty clean. It doesn't hurt to clamp it down a bit, though. I have it pointed to a non-replica copy of the DB for now, since I plan to do some destructive tinkering with the data when I actually make a UI that I don't want replicating back to "production" just yet, but I'll clear the Darwino side and replicate in the live data when I'm ready.

To enable replication in the Darwino app, I commented out the labeled block in the JEE project's web.xml and set a few properties in my Tomcat server's darwino.properties (to externalize sensitive information):

frostillicus_blog.sync-enabled=true
frostillicus_blog.sync-url=http://pelias-l.frostillic.us/darwino.sync

Once that was set, I launched the app using a Tomcat instance in Eclipse, and I could see it doing its thing:

Start deploying database frostillicus_blog, POSTGRESQL, 0/0
Finished deploying database frostillicus_blog, 0, 0/0
Start replication with server http://pelias-l.frostillic.us/darwino.sync
Started replication Pull frostillicus_blog, estimated entries: 5013 [September 15, 2016 2:03:37 PM EDT]
1 processed, 0% (total 5013, remaining time 1m55s, avg rate 1/s, instant rate 0/s)
664 processed, 13% (total 5013, remaining time 13s, avg rate 332/s, instant rate 332/s)
1313 processed, 26% (total 5013, remaining time 11s, avg rate 328/s, instant rate 324/s)
1941 processed, 38% (total 5013, remaining time 9s, avg rate 323/s, instant rate 314/s)
2518 processed, 50% (total 5013, remaining time 7s, avg rate 314/s, instant rate 288/s)
3086 processed, 61% (total 5013, remaining time 5s, avg rate 308/s, instant rate 284/s)
3597 processed, 71% (total 5013, remaining time 4s, avg rate 299/s, instant rate 255/s)
4069 processed, 81% (total 5013, remaining time 2s, avg rate 290/s, instant rate 236/s)
4489 processed, 89% (total 5013, remaining time 1s, avg rate 280/s, instant rate 210/s)
4929 processed, 98% (total 5013, remaining time 0, avg rate 273/s, instant rate 220/s)
5013 processed, 100% (total 5013, remaining time 0, avg rate 278/s, instant rate 84/s)
+++ Finished, 5013 processed (estimated 5013, time 18s, avg rate 278/s)

Once that was done, I went over to the default utilitarian web UI to make sure everything looked good, and it did:

(the advice about reverse proxies still stands, by the way)

Nothing app-specific in there yet (and I haven't hooked up authentication to Domino yet, so I don't have my Gravatar icon), but it shows that the data made the trip none the worse for wear, authors field and all.

That will do it for the initial phase. I plan to revisit this down the line, once I've made a decision on a UI framework and have the time to actually start implementing that. That will be an interesting one - there are strong reasons to make single-page applications in JavaScript, but my heart is still in server-side toolkits. That will be a choice for another day, though.


* If you'll forgive the blatant sales pitch.

There Are Pods That Need Casting

  • Sep 6, 2016

I had the pleasure of taking part in the long-awaited demise episode of This Week In Lotus!

http://thisweekinlotus.com/115-doing-a-three-way/

We covered a lot of ground, with a reflection on this year's MWLUG, the Sturm und Drang about Domino, Toscana/IBM design, and a few other topics. Most important, though, is the special announcement at the end of the episode, which you'll just have to listen for. Or skip to the end, I guess; there's a tracking bar right on the page.

Quick XPages Utility: Keep Alive and Alert

  • Aug 30, 2016

For one of my projects recently, I put together a small custom control that I sure wish I had thought to do years ago: a "keep alive and alert" control. For a long time now, the Extension Library has had a "keepAlive" control, which keeps a page session (and, usually, authentication) alive while the user has the browser window open, avoiding the otherwise-common issue of someone sitting on a page "too long" and having it break underfoot. However, that doesn't cover the edge cases: the user putting their computer to sleep and waking it with the same page open, the server rebooting, a server cluster failover, expiring SSO token, or so forth. In those cases, the problem will fall more or less "silently" to the JavaScript console, and the page will just be mysteriously unresponsive.

This control has a similar starting point, where it will ping the server periodically (every 10 seconds in this case), but will also display a Bootstrap modal alert when things go awry. It's not too picky about the cause of the problem: since usually the only practical solution is to reload the page, it just says that and leaves it there. This could also be redone to be more efficient like the ExtLib one (which would avoid the page recomputation inherent in the partial refresh), and it may not cover the case of authorization expiring in an app that allows anonymous access, but it should do the job nicely in the normal case:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">
	<xp:div id="keepAliveAndAlert" style="display: none"></xp:div>
	<xp:scriptBlock><xp:this.value><![CDATA[
		window.__keepAliveAndAlert = setInterval(function() {
			XSP.partialRefreshGet("#{id:keepAliveAndAlert}", {
				onComplete: function() {
					// Good!
				},
				onError: function() {
					jQuery(dojo.byId("#{id:keepAliveAndAlertError}")).modal({
						backdrop: 'static'
					});
					
					clearInterval(window.__keepAliveAndAlert);
					window.__keepAliveAndAlert = null;
				}
			});
		}, 10 * 1000);
	]]></xp:this.value></xp:scriptBlock>
	<xp:div style="display: none" id="keepAliveAndAlertError" styleClass="modal fade error-modal" role="dialog">
		<div class='modal-dialog'><div class='modal-content'>
			<div class='modal-header'><h4 class='modal-title'>Page Session Expired</h4></div>
			<div class='modal-body'>
				<p>Your page session has expired.</p>
				<p>Please&#160;<a href="javascript:location.reload()">reload</a>&#160;the page to continue working.</p>
			</div>
		</div></div>
	</xp:div>
</xp:view>

Change Is In The Air

  • Aug 26, 2016

During last week’s MWLUG, there was a clear sense that things are a little different this year. Dave Navarre dubbed the technical implications “platform agnosticism”, while I geared my presentation towards the feeling that change is in the air.

This is not totally new. Red Pill Now cast aside the XPages UI layer and most of the assumptions of Domino development to move to a new level; PSC's presentations have long developed a polyglot tone, and this ramped up this year; and people like Paul Withers have been growing with tools like Vaadin.

It's not too important to dive into the specifics of market forces or the shifting sands of technology, and the platform defensiveness that we tend to wear as a cozy blanket doesn't serve anyone properly. Our beloved Domino app-dev platform has grown pretty long in the tooth and it doesn’t seem like it’s in for a revitalization.

The situation is, fortunately, an opportunity.

One of the things I hoped to convey in my presentation is that, though the prospect of learning some of the ever-changing array of modern development tools is daunting, it is also exhilarating and profitable both professionally and personally. As insane as the tangled web of server frameworks, JS optimizers, language transpilers, automation tools, dependency managers, and so forth may seem, particularly compared to the simple days of Notes client development, there is a great deal of good news.

The popular tools are awash in documentation, with clear examples for doing basically whatever you will want to do. There's also a lot of conceptual overlap and familiarity: if your tool of choice loses favor, it won't be a start from square one to learn the next. And it's not required to learn every single thing that comes along. If you build yourself a Java web stack using, say, Spring that does the job, it's not also necessary to learn every single new client-side JS app framework that comes along.

And, in the mean time, there's a lot of great work left to do with Domino. There are XPages applications in use and development, and these will go a very long way. Domino remains a very capable platform, and the path through XPages can be a very natural lead-in to other technologies, especially if you focus on the aspects that carry on: Java, data separation, REST services, and so forth.

For my part, I, too, still have great work to do on Domino and XPages, but I'm also expanding beyond it with eyes open. As I mention frequently, I believe that Darwino is the best path for a number of reasons. When I have the opportunity, I plan to start getting into the meat-and-potatoes reasons why and examples of how to actually use the thing. Time permitting, I hope to have a series at some point for converting my blog over to a Darwino+JEE application, and I'll share my process of picking my tools and replicating with its current NSF form as I go. If all goes well, it should serve as an example of taking an existing XPages app and transforming it into something new.

This is an opportunity, and it's an exciting time.


Postscript: This is the optimistic take, granted, and some people’s situations are a bit more dire. Admins, I imagine, are in a strange spot (I hope you’ve been brushing up on ancillary tools!), and a lot of companies are doing a lot of hand-wringing about the future for app dev and maintenance as well, particularly those with a heavy Notes-client dependency. My point is that it’s not necessary to get too mired in the doom and gloom.

MWLUG 2016 Slides

  • Aug 20, 2016

I just returned from this year's MWLUG, held in surprisingly-rainy Austin, Texas. As every year, MWLUG is an outstanding event, particularly for a development crowd (though I think admins get tons of material too). I'm definitely looking forward to next year's in DC, and not merely because that's a pretty quick drive for me.

I'll have some more to write later to follow up on the themes of this year's event, but in the mean time I've uploaded the slides from my presentation:

AD106 - Expand Your Apps And Skills To The Wider World

This was a fun one to present, and should lead into a lot of good blog material in the future.

MWLUG 2016

  • Aug 15, 2016

MWLUG 2016 is happening this week, down in I'm-sure-it's-not-deathly-hot Austin, Texas. MWLUG has really proven to be a top-tier conference for our community, and I'm looking forward to it again this year. To be fair, part of that is that I'll be presenting on Friday, with a session entitled "Expand Your Apps And Skills To The Wider World":

The technological world has changed, but the work done by your Domino apps has not. Rather than discarding your old applications in a distruptive process, it is possible to extend them forward into the modern world: native mobile applications, modern web technologies, and integration with new platforms. This session will demonstrate using active open-source technologies to build new, mobile-enabled UIs, run smoothly on Bluemix, integrate with Watson, perform deep reporting with DashDB, and integrate authentication in Connections Cloud. These examples use Darwino for Domino connectivity and app structure, and this session will also introduce the forthcoming Darwino.org open-source community.

This will be something of a followup to my previous sessions about becoming familiar first with Java in XPages and then eventually with Maven. It's an interesting time to be in the spot we are, and I hope to make some sense of one of the paths we can take.

So, please join me Friday at 11:30 - 12:30 (where hopefully we won't be too ravenous for lunch) and all week to catch up and talk shop.

Release Weekend: ODA and Darwino

  • Aug 2, 2016

This past weekend was a nice one for releases to a couple of the projects I work on: the OpenNTF Domino API and Darwino.

The ODA release is something of a "consolidation" release over 2.0.0: it fixes a few of the bugs that cropped up since then, adds some important lower-level tweaks, and brings the graph REST API into the mainline release. One note with the REST API is that, due to making use of a recently-added extension to the core code, it requires a recent release of the Extension Library, 9.0.1_17 or higher.

The Darwino release is something of a preparatory release, containing some significant improvements since the initial 1.0 earlier this year. A lot of the features will warrant a proper announcement post, but the ones I find most intriguing (or worked on the most) are significantly-improved Domino replication, a new scheduling framework, and some nice cloud-deployment improvements, such as Watson integration tools and proper support for Microsoft Azure. This also sets the ground for a number of features in the next major release, which exist in initial form now, but need some final polish.

Should I have time (it continued to be a very-busy summer), both of these will warrant some more discussion. But, in the mean time, give the new versions a shot!

Provisions for the Journey

  • Jun 19, 2016

In my last post, I ended up recommending that Domino developers and administrators take some time, if at all possible, to dive into new tools. In some cases, that may be in order to stay indefinitely, but, even otherwise, learning a new environment would have tremendous benefit to your existing Domino work.

Since I'm not a full-time administrator myself, my advice is admittedly a bit thin. It mostly involves my oft-repeated suggestion to try out a reverse-proxy setup for HTTP. That could parlay into use with mail as well, but even just becoming more familiar with a multi-tiered setup and the essentially-standard OpenSSL tools for certificate manipulation would pay dividends.

For developers, there are a great many potential paths to walk, and I think it's extremely useful to at least take one step down each. The use of that reminds me of the "Programming Languages" course I took in college. It involved rapid-fire coverage of C, C++, Scheme, and Prolog - four distinct languages with pretty distinct programming styles (the latter two moreso than the former two), meant to introduce students to the different notions of programming. I don't use Lisp or (lord knows) Prolog in my day-to-day programming life, but having learned a little bit about functional and logic programming at the time has paid dividends over the years.

There are a couple broad topics I can think of that are worth investigating, including ones that we as a community have already been working on.

Source Control

For the most part, the importance of source control has basically sunk in as a "we should do this" sort of thing in the Domino community. However, the inconvenience associated with maintaining an on-disk project means we're not all the best at doing this religiously. And, indeed, we're often hampered by both old habits and Domino's licensing terms that prevent us from doing this properly: there's a strong strain of "multiple people working on the same NSF" in Domino's history, and that doesn't mix well with source control (or XPages, for that matter).

So I recommend not just knowing about source control, but also trying (as much as licensing allows) to use this as an opportunity to switch to a more "traditional" model for app development. I like to think of this type of development as equivalent to working on a desktop app: it would be madness to have multiple people working in the same Xcode/VS/etc. workspace on the same binary build, and then particularly trying to have an end user also use that resultant binary. Instead, it's better to have each developer work separately on distinct non-replica NSFs, ideally on local dev servers, and then deploy the changes as an NTF.

Java EE WARs

Java EE in general is a very worthwhile target for interest, especially because it encompasses so many other possibilities. But I'd like to call attention in particular to the WAR file model, which is roughly equivalent to an NSF for app deployment.

When I first learned web programming, with PHP, it followed the "just toss a bunch of files on disk and visit their paths in a web browser" model, which basically worked, but taught some very bad habits. Domino improves on this a bit, with everything in its place in an NSF, but still suffers from a bit of the same malady: the "filesystem" of design elements is what is exposed to the browser directly, URL-wise. In a more fully-app-based system, though, there's a lot more flexibility about how the incoming URLs map to the underlying code and resources, and Java EE's encapsulated nature plays on this well.

With a WAR file, you have the "app container" advantage of an NSF but also gain the "everything after the app name is mine" benefit of a controlled URL router, and that shift helps conceptualize apps as something of a "mount point" onto an app server. So you may have a blog app that you load up as "/blog" on your server, and then that app can control the traffic beneath it, but is similarly a bit walled off from everything above it. It's the sort of "feel" that settle in with time, and then it feels weird to lack that flexibility.

This model also fits particularly well into app platforms like BlueMix or Heroku.

Testing

This one's difficult for Domino developers! No matter the tooling used for an NSF, being able to dive into automated testing is difficult. You can coax Maven into running some tests, though that will lack an XPages environment. You can test on the XPages side with JUnit4Xpages, though that won't be part of an automated build without more hoops. There's no real equivalent to the type of fully-fleshed-out testing tools available for other platforms now.

So that's lead to unit/integration/whatever testing not even being on the horizon for most Domino developers. That's particularly a shame because it's already something of a chore to get into the testing habit even when the tools do everything they can to help you. It's essentially like brushing your teeth when you're a kid: you do it because you're told to, but it's not something you'd ever do of your own accord.

It's worth it, though! It will probably require working on a fully non-Domino project, but going through the steps of building a fleshed-out system with automated tests along the way is a valuable experience. The goal is to get to the point where it feels weird to not have tests, much in the same way as lacking source control - an untested change should feel like it doesn't "count". It's a big world, too: beyond just small unit tests, there's tooling to run a web app in a test environment and run user interactions, or do the same on mobile-device simulators.

Build Servers

This dovetails with testing, in that one of the primary benefits of having a build server like Jenkins or Bamboo is that they can act as a neutral, clean environment to build the whole app and run the tests, informing you of any errors or failures. This is another thing that's made much more difficult by Domino, though it's possible.

Beyond testing, having a build server reaps benefits with project organization and delivery pipelines. Jenkins's recent updates have focused on that sort of thing, where the tool will help you manage pulling down the latest code from SCM, building it, and then deploying the results to dev or live servers, as desired. Build servers won't necessarily change the way you develop, but they will prove to be a faithful companion, with more uses the more you get comfortable with them.

Unix

Get comfortable with the command line. Other than Microsoft's tooling, which has a very schizophrenic relationship with the CLI, basically all modern development environments swim in the Unix world. Not everything has to be done via CLI - certainly, it's much more comfortable to use SourceTree than to do all git commands manually - but it's hard to find a tutorial that doesn't have at least a few tossed-off terminal commands. Mac- and Linux-based developers are at a distinct advantage with this, but Windows has Cygwin and the ported Linux userland in the next Windows 10 release.

Along this line, it's tremendously useful to have a bit of Linux server admin knowledge. I recommend either spinning up a Linux VM locally (Ubuntu has served me well) or using Linode. Linode in particular is so cheap and convenient that it's very much worth tossing them a couple bucks to run a server for a little while with full confidence that you can melt it into the ground without consequence. The delightful iOS-focused podcast Under the Radar had an episode on this topic a while back that's worth a listen.

You should probably also learn a little about vi.

MVC

Back in the realm of pure programming knowledge, it's extremely valuable to learn about MVC and related concepts. XPages sort of brought this to the Domino world, but it suffered from lack of wholehearted adoption and its legacy baggage made it very difficult to really go full MVC even if you wanted to.

The good news is that essentially every other active framework encourages or enforces this approach. When diving into one, it can be a bit jarring at first - there's a lot of "ugh, can't I just put the code here?" going on - but it falls into place very quickly.

So where to start?

Picking a place to start is a daunting prospect, given the ever-shifting sands of the modern development world.

One of my favorite tutorials for a web framework is Michael Hartl's Rails tutorial, which is kindly available for free online (with options to purchase other forms). It not only provides an up-to-date introduction to Rails development, but also covers a lot of the other ground mentioned above: the command line, Heroku, Git, and test-driven development.

In another direction, I recommend taking a look at Meteor, which is something of an all-in approach to JavaScript-based development. It does an interesting job blurring the lines between client and server and makes a very compelling case for "JavaScript everywhere".

Beyond that, really just learning any development toolchain will help. There's not too much need to sweat picking "the right one" at each step - any knowledge gained in one will help in others, either directly or by giving you some additional perspective. The most important thing, especially now, is learning something.

Change Bitterness and Accidents of History

  • Jun 17, 2016

It's pretty easy to see that change is in the air for Domino types. It's been taking a number of forms for a while now - the long delay since the release of 9.0.1 and associated aging of the tools and infrastructure have led to a series of forced adaptations for developers and administrators. Developers, for example, have had to keep light on their feet to adapt to new browsers and devices that the framework doesn't automatically support, as well as a shift toward manually including jQuery and other tools that have a bit more wind at their back than Dojo. Administrators, for their part, have had a series of heart attacks related to SSL and other security matters, usually involving a a lot of noise followed by (unfortunately, I feel) a good-enough patch from IBM.

That sort of thing isn't likely to get any smoother. In large part, that's entirely distinct from anything IBM does: the genie's out of the bottle when it comes to fast-moving platforms, and the best we can hope for is a sort of still-moving linga franca that can be sort-of-stable on the majority of targets. But then part of that is Domino: for all its virtues, it hasn't adapted for the modern world, no matter how much some of us would have liked it to. And that's had some negative side effects on us as a customer base, side effects that manifest as a gut-reaction rejection of the modern ways of doing things.

Take the SSL thing, one of my soap boxes: though the immediate problem could be summarized as "IBM should update their SSL stack", the larger issue it exposed was that our beloved monolith is mortally vulnerable to a single component falling behind. And, in fact, it made clear that we're spoiled by the approach: many of the reactions were basically that we shouldn't have to worry about things like reverse proxies or the general notion of distinct systems for web front-ends and the app server. And that initial rejection of the hassle has implications, limiting the average Domino installation's capacity to scale for load balancing or failover in a smooth way, things that come almost "for free" with a reverse-proxied setup common among almost all other app servers.

Developers have it worse: the "power user turned developer" history of Notes and the particular peculiarities of the platform* have left us desperately behind the baseline for modern development. The tooling has coddled us into preferring inline scripts and procedural programming to structured code organization, into viewing a thoroughly-staid language like Java as something outrageously complex, of only begrudgingly adopting SCM due to the platform-induced hassle, and of almost entirely ignoring automated testing. And similarly to the SSL discussion, the historic "one tool for every job" nature of Domino leads to natural pushback when faced with other platforms.

And I get why! And I feel it too. It's a real PITA to now always have to be on the lookout for some new point release of iOS or Android to break drop-down boxes or something, as opposed to years of deploying Notes apps that looked and worked identially across every version forever (pretty much). It's also a drag to run into situations where the problem is "easy" on Domino but more cumbersome elsewhere, like platforms that farm out their FT indexes to distinct servers, or don't include document/record-based security. That makes it very easy to become blinded to the tradeoffs, though. It may be nice that Domino is a one-stop-shop for so much, but it's a shop that requires Designer, that makes it very difficult to use third-party-dependency systems like Maven (even within Maven projects), that lags in DB features found elsewhere, that is only awkwardly accessible from other app-dev frameworks, that has an API that's a bit older than Windows Me, and that essentially never showed up for the modern development conversation.

Domino has always had a lot to recommend it, and XPages has carried us very far. And hey, this is enterprise software - even if there's never a major new version, there'll be paying work forever. It just may not be the kind of work you want to do, and it is almost definitely not truly healthy for the companies paying for it. So what I recommend is that you have a plan. The good news is that there are a great many next steps that build smoothly on existing Domino knowledge and, potentially, infrastructure. Certainly, I'm thoroughly biased in the direction of Darwino, but that's one of many. You could also do worse than learning a mature platform like Ruby on Rails (heck, you could run that on JRuby on Tomcat or WebSphere). Take some time to learn about reverse proxies and modern web-server setups. Basically: something. Just do it with an open mind, and don't balk at the first thing that's more complicated than the Domino equivalent. I think a bit of that will serve you very well.


* And, to be fair, of the overly-conservative nature of enterprise programming.

Other than, of course, being one of the progenitor NoSQL databases.

Code Safety and Pedantry

  • Jun 3, 2016

Lately, I've been musing a lot on the topic of code "correctness" - that is, beyond the normal case of wanting code to do what I intended, and further into the realm of sweating even extremely-miniscule details. A lot of this is due to my continued watching of the evolution of Apple's Swift language (I highly recommend following Erica Sadun's blog for this). Swift is very much in the camp of "make sure all your 'i's are dotted and 't's crossed" languages, as opposed to more fast-and-loose languages like JavaScript or Ruby.

I've gone back and forth on the overarching concepts from time to time. I've long been a big Ruby fan, and a lot of that is because of a general feeling that, if you let go of a lot of the "strict old aunt of a compiler" restrictions, you gain a tremendous amount of expressiveness and productivity with few real-world problems. On the other hand, being immersed in Java all the time has shifted my brain to appreciating the benefits of stronger compile-time checks (at least on paper). Overall, I'm more on the latter side than the former now, double-edged sword though it is. This is why I've been diving into things like aggressive null analysis in my code. Even when it seems like it's being a pedant, there are certain classes of bugs that it finds that I wouldn't even normally think of on the fly. For example, the null checker flags this as being a potential NPE:

if(this.foo != null) {
	this.foo.doSomething();
}

My first reaction upon seeing that was along the lines of "you're full of crap, Eclipse", but then I noticed the small path a bug could take to creep in: multithreading. If I'm in a situation where the object containing foo is used across threads, there's a possibility where Thread A would evaluate this.foo != null to true and start to step in to the block. Then, Thread B would get its turn on the processor and evaluate this.foo = null in another method. Thread A would then pick up and try to call doSomething() on the newly-minted null. So I've swallowed my pride and started writing safer code like:

SomeObject localFoo = this.foo;
if(localFoo != null) {
	localFoo.doSomething();
}

What I've always admired about Swift is that it takes these sorts of lessons to heart and adapts the syntax to suit. My interest in code-correctness pedantry in Java has led me to write out verbose abominations like this:

private final @NotNull String foo;

Three of the conceptual tokens there are purely to say things that are best practices to start with: I don't want this property accessible outside the class, I want to make sure it's assigned during construction and not change thereafter, and I want to ensure it's not null. The Swift variant is:

let foo: String

Same thing, half the typing. And, as a bonus, since nullability checking is built in to the language and not a by-convention thing like the null annotations in Java, I can be sure that the rules will be applied. That sort of thing is the dream! But, since Java is the best language for the work I'm doing for now, the important thing is that it at least suits, verbose or not. In a lot of languages, it gets much more difficult to have this sort of assurance.

So, hassle as it is, I suggest that other Domino developers, on their paths through Java, consider picking up the same habits. For every time you run into something like Java complaining that it can't convert a List<String> into a List<Object>, diving fully into null checks and immutability will save you a late-night crash report and angry user. As you develop more in Java, give it a try.

Darwino for Domino: Conceptual Overlap and Distinctions

  • Jun 1, 2016

I've talked a bit so far about how Darwino related to Domino from a development perspective, but I think it'd also be useful to delve into specifically which concerns the two platforms address, to see where they overlap and where they don't.

There are two main categories to cover, since Darwino inherits Domino's unusual trait of pouring over from "database" to "app-dev platform".

Database

As I covered a few posts ago, the two are similar at a conceptual level, both being replicating document databases with document-level access control. Aside from the difference between an NSF note's format and a JSON document, the main distinction is that Darwino doesn't cover the actual physical storage of data. Instead, it is based on top of existing SQL servers of various stripes (PostgreSQL and SQLite being the most common). A Darwino application creates a series of tables and uses them as the backing store for the conceptual document database.

This has a number of implications. The main one is that there isn't a "Darwino server" as such - instead, there are SQL databases and Darwino applications acting in tandem. In developing an application, this isn't generally a concern: the Darwino APIs are the same across each database, in the same sort of way that a Domino application doesn't care about the ODS version. However, being backed by a SQL server has some distinct advantages: the server can be administered and optimized using the same knowledge you would use for a "normal" SQL-backed app, and the ability of modern DBs to index JSON data opens up a world of possibilities (think NSFDB2, except good).

The flip side of this bleeds into the second category, as it means that a Darwino application consists of at least two parts: the SQL database and the application, which veers from Domino's "everything in one package" promise slightly.

Application

Things diverge most significantly (though at least as promisingly) when it comes to the application level. Domino has a few "official" ways to develop applications (Notes, legacy web, and XPages) and then hooks to sort of act like a Java EE server, albeit with some notable limitations. Darwino, on the other hand, exists as a sort of "glue layer" in between the database and the application: lower-level than XPages but higher-level than just a database driver.

Darwino provides a common platform for writing Java-based applications, with various services for managed beans, user directories, and so forth, written to work consistently across all of the platforms it targets. Again, this starts out similar to Domino, but diverges in the areas where Darwino takes advantage of other technologies.

At the low level, since Darwino's main requirement is "a Java runtime", it is able to run smoothly on various Java EE servers, on Android, on iOS, and on pretty much anything that provides a capable-enough Java environment (such as, say, Domino). It also, incidentally, means that it works great on Java 8.

At the high level, Darwino doesn't prescribe a specific UI framework, so the field is open to use any of the tremendous array of rapidly-developing Java frameworks on the web side and, as desired, native UI toolkits on mobile. There's a bit of an inherent bias towards REST+client JS applications, since then the same code entirely can be used on both web and mobile (as not every Java web tooklit works on the mobile mini web server as it is now), but that's not obligatory.

Overall

So the overall idea is that Darwino doesn't solve every problem that Domino does, but the problems it chooses to farm out are in the areas where that brings tremendous benefit. In each area where Darwino uses third-party support, it benefits from the tremendous advancements made in recent years, without requiring jumping through weird hoops to get modern techniques to work.

Speaking at Social Connections Toronto

  • May 30, 2016

By virtue of one of the original speakers having to cancel, I will be presenting at Social Connections in Toronto next week! Specifically, it will be on the 7th at 2:30, and the topic will be OpenNTF's new tooling and initiatives:

OpenNTF has long been a central hub for community and open-source development in the Notes/Domino world. In the past year, however, it has also expanded its capabilities with new tooling and a broadening scope to the larger IBM portfolio. This presentation will discuss OpenNTF’s history and its plans for the future, including a look at its newly-launched suite of source control, issue tracking, and continuous-delivery tools.

So, if you're attending Social Connections, drop on by the Toronto Room on Tuesday afternoon.

Old App Idea: App Manager

  • May 24, 2016

Years back, I took a small whack at an idea that had been percolating in my head: a "app manager" application that would assist with the XPage-specific portions of running a Domino server. It would cover a lot of ground that Administrator really doesn't touch, such as inspecting NSFs to see which contain XPage artifacts, highlighting potential problems with them, and assisting with app-design backup and deployment.

It never really got too far, since there are always a hundred other things to work on, but I always liked the idea. I decided to toss what code I had laid down up on GitHub:

https://github.com/jesse-gallagher/app-manager

I don't imagine I'll really have time to build on that unless I'm suddenly really struck with inspiration. But either way, the code is there for anyone interested in taking a look.