Change Is In The Air

  • Aug 26, 2016

During last week’s MWLUG, there was a clear sense that things are a little different this year. Dave Navarre dubbed the technical implications “platform agnosticism”, while I geared my presentation towards the feeling that change is in the air.

This is not totally new. Red Pill Now cast aside the XPages UI layer and most of the assumptions of Domino development to move to a new level; PSC's presentations have long developed a polyglot tone, and this ramped up this year; and people like Paul Withers have been growing with tools like Vaadin.

It's not too important to dive into the specifics of market forces or the shifting sands of technology, and the platform defensiveness that we tend to wear as a cozy blanket doesn't serve anyone properly. Our beloved Domino app-dev platform has grown pretty long in the tooth and it doesn’t seem like it’s in for a revitalization.

The situation is, fortunately, an opportunity.

One of the things I hoped to convey in my presentation is that, though the prospect of learning some of the ever-changing array of modern development tools is daunting, it is also exhilarating and profitable both professionally and personally. As insane as the tangled web of server frameworks, JS optimizers, language transpilers, automation tools, dependency managers, and so forth may seem, particularly compared to the simple days of Notes client development, there is a great deal of good news.

The popular tools are awash in documentation, with clear examples for doing basically whatever you will want to do. There's also a lot of conceptual overlap and familiarity: if your tool of choice loses favor, it won't be a start from square one to learn the next. And it's not required to learn every single thing that comes along. If you build yourself a Java web stack using, say, Spring that does the job, it's not also necessary to learn every single new client-side JS app framework that comes along.

And, in the mean time, there's a lot of great work left to do with Domino. There are XPages applications in use and development, and these will go a very long way. Domino remains a very capable platform, and the path through XPages can be a very natural lead-in to other technologies, especially if you focus on the aspects that carry on: Java, data separation, REST services, and so forth.

For my part, I, too, still have great work to do on Domino and XPages, but I'm also expanding beyond it with eyes open. As I mention frequently, I believe that Darwino is the best path for a number of reasons. When I have the opportunity, I plan to start getting into the meat-and-potatoes reasons why and examples of how to actually use the thing. Time permitting, I hope to have a series at some point for converting my blog over to a Darwino+JEE application, and I'll share my process of picking my tools and replicating with its current NSF form as I go. If all goes well, it should serve as an example of taking an existing XPages app and transforming it into something new.

This is an opportunity, and it's an exciting time.


Postscript: This is the optimistic take, granted, and some people’s situations are a bit more dire. Admins, I imagine, are in a strange spot (I hope you’ve been brushing up on ancillary tools!), and a lot of companies are doing a lot of hand-wringing about the future for app dev and maintenance as well, particularly those with a heavy Notes-client dependency. My point is that it’s not necessary to get too mired in the doom and gloom.

MWLUG 2016 Slides

  • Aug 20, 2016

I just returned from this year's MWLUG, held in surprisingly-rainy Austin, Texas. As every year, MWLUG is an outstanding event, particularly for a development crowd (though I think admins get tons of material too). I'm definitely looking forward to next year's in DC, and not merely because that's a pretty quick drive for me.

I'll have some more to write later to follow up on the themes of this year's event, but in the mean time I've uploaded the slides from my presentation:

AD106 - Expand Your Apps And Skills To The Wider World

This was a fun one to present, and should lead into a lot of good blog material in the future.

MWLUG 2016

  • Aug 15, 2016

MWLUG 2016 is happening this week, down in I'm-sure-it's-not-deathly-hot Austin, Texas. MWLUG has really proven to be a top-tier conference for our community, and I'm looking forward to it again this year. To be fair, part of that is that I'll be presenting on Friday, with a session entitled "Expand Your Apps And Skills To The Wider World":

The technological world has changed, but the work done by your Domino apps has not. Rather than discarding your old applications in a distruptive process, it is possible to extend them forward into the modern world: native mobile applications, modern web technologies, and integration with new platforms. This session will demonstrate using active open-source technologies to build new, mobile-enabled UIs, run smoothly on Bluemix, integrate with Watson, perform deep reporting with DashDB, and integrate authentication in Connections Cloud. These examples use Darwino for Domino connectivity and app structure, and this session will also introduce the forthcoming Darwino.org open-source community.

This will be something of a followup to my previous sessions about becoming familiar first with Java in XPages and then eventually with Maven. It's an interesting time to be in the spot we are, and I hope to make some sense of one of the paths we can take.

So, please join me Friday at 11:30 - 12:30 (where hopefully we won't be too ravenous for lunch) and all week to catch up and talk shop.

Release Weekend: ODA and Darwino

  • Aug 2, 2016

This past weekend was a nice one for releases to a couple of the projects I work on: the OpenNTF Domino API and Darwino.

The ODA release is something of a "consolidation" release over 2.0.0: it fixes a few of the bugs that cropped up since then, adds some important lower-level tweaks, and brings the graph REST API into the mainline release. One note with the REST API is that, due to making use of a recently-added extension to the core code, it requires a recent release of the Extension Library, 9.0.1_17 or higher.

The Darwino release is something of a preparatory release, containing some significant improvements since the initial 1.0 earlier this year. A lot of the features will warrant a proper announcement post, but the ones I find most intriguing (or worked on the most) are significantly-improved Domino replication, a new scheduling framework, and some nice cloud-deployment improvements, such as Watson integration tools and proper support for Microsoft Azure. This also sets the ground for a number of features in the next major release, which exist in initial form now, but need some final polish.

Should I have time (it continued to be a very-busy summer), both of these will warrant some more discussion. But, in the mean time, give the new versions a shot!

Provisions for the Journey

  • Jun 19, 2016

In my last post, I ended up recommending that Domino developers and administrators take some time, if at all possible, to dive into new tools. In some cases, that may be in order to stay indefinitely, but, even otherwise, learning a new environment would have tremendous benefit to your existing Domino work.

Since I'm not a full-time administrator myself, my advice is admittedly a bit thin. It mostly involves my oft-repeated suggestion to try out a reverse-proxy setup for HTTP. That could parlay into use with mail as well, but even just becoming more familiar with a multi-tiered setup and the essentially-standard OpenSSL tools for certificate manipulation would pay dividends.

For developers, there are a great many potential paths to walk, and I think it's extremely useful to at least take one step down each. The use of that reminds me of the "Programming Languages" course I took in college. It involved rapid-fire coverage of C, C++, Scheme, and Prolog - four distinct languages with pretty distinct programming styles (the latter two moreso than the former two), meant to introduce students to the different notions of programming. I don't use Lisp or (lord knows) Prolog in my day-to-day programming life, but having learned a little bit about functional and logic programming at the time has paid dividends over the years.

There are a couple broad topics I can think of that are worth investigating, including ones that we as a community have already been working on.

Source Control

For the most part, the importance of source control has basically sunk in as a "we should do this" sort of thing in the Domino community. However, the inconvenience associated with maintaining an on-disk project means we're not all the best at doing this religiously. And, indeed, we're often hampered by both old habits and Domino's licensing terms that prevent us from doing this properly: there's a strong strain of "multiple people working on the same NSF" in Domino's history, and that doesn't mix well with source control (or XPages, for that matter).

So I recommend not just knowing about source control, but also trying (as much as licensing allows) to use this as an opportunity to switch to a more "traditional" model for app development. I like to think of this type of development as equivalent to working on a desktop app: it would be madness to have multiple people working in the same Xcode/VS/etc. workspace on the same binary build, and then particularly trying to have an end user also use that resultant binary. Instead, it's better to have each developer work separately on distinct non-replica NSFs, ideally on local dev servers, and then deploy the changes as an NTF.

Java EE WARs

Java EE in general is a very worthwhile target for interest, especially because it encompasses so many other possibilities. But I'd like to call attention in particular to the WAR file model, which is roughly equivalent to an NSF for app deployment.

When I first learned web programming, with PHP, it followed the "just toss a bunch of files on disk and visit their paths in a web browser" model, which basically worked, but taught some very bad habits. Domino improves on this a bit, with everything in its place in an NSF, but still suffers from a bit of the same malady: the "filesystem" of design elements is what is exposed to the browser directly, URL-wise. In a more fully-app-based system, though, there's a lot more flexibility about how the incoming URLs map to the underlying code and resources, and Java EE's encapsulated nature plays on this well.

With a WAR file, you have the "app container" advantage of an NSF but also gain the "everything after the app name is mine" benefit of a controlled URL router, and that shift helps conceptualize apps as something of a "mount point" onto an app server. So you may have a blog app that you load up as "/blog" on your server, and then that app can control the traffic beneath it, but is similarly a bit walled off from everything above it. It's the sort of "feel" that settle in with time, and then it feels weird to lack that flexibility.

This model also fits particularly well into app platforms like BlueMix or Heroku.

Testing

This one's difficult for Domino developers! No matter the tooling used for an NSF, being able to dive into automated testing is difficult. You can coax Maven into running some tests, though that will lack an XPages environment. You can test on the XPages side with JUnit4Xpages, though that won't be part of an automated build without more hoops. There's no real equivalent to the type of fully-fleshed-out testing tools available for other platforms now.

So that's lead to unit/integration/whatever testing not even being on the horizon for most Domino developers. That's particularly a shame because it's already something of a chore to get into the testing habit even when the tools do everything they can to help you. It's essentially like brushing your teeth when you're a kid: you do it because you're told to, but it's not something you'd ever do of your own accord.

It's worth it, though! It will probably require working on a fully non-Domino project, but going through the steps of building a fleshed-out system with automated tests along the way is a valuable experience. The goal is to get to the point where it feels weird to not have tests, much in the same way as lacking source control - an untested change should feel like it doesn't "count". It's a big world, too: beyond just small unit tests, there's tooling to run a web app in a test environment and run user interactions, or do the same on mobile-device simulators.

Build Servers

This dovetails with testing, in that one of the primary benefits of having a build server like Jenkins or Bamboo is that they can act as a neutral, clean environment to build the whole app and run the tests, informing you of any errors or failures. This is another thing that's made much more difficult by Domino, though it's possible.

Beyond testing, having a build server reaps benefits with project organization and delivery pipelines. Jenkins's recent updates have focused on that sort of thing, where the tool will help you manage pulling down the latest code from SCM, building it, and then deploying the results to dev or live servers, as desired. Build servers won't necessarily change the way you develop, but they will prove to be a faithful companion, with more uses the more you get comfortable with them.

Unix

Get comfortable with the command line. Other than Microsoft's tooling, which has a very schizophrenic relationship with the CLI, basically all modern development environments swim in the Unix world. Not everything has to be done via CLI - certainly, it's much more comfortable to use SourceTree than to do all git commands manually - but it's hard to find a tutorial that doesn't have at least a few tossed-off terminal commands. Mac- and Linux-based developers are at a distinct advantage with this, but Windows has Cygwin and the ported Linux userland in the next Windows 10 release.

Along this line, it's tremendously useful to have a bit of Linux server admin knowledge. I recommend either spinning up a Linux VM locally (Ubuntu has served me well) or using Linode. Linode in particular is so cheap and convenient that it's very much worth tossing them a couple bucks to run a server for a little while with full confidence that you can melt it into the ground without consequence. The delightful iOS-focused podcast Under the Radar had an episode on this topic a while back that's worth a listen.

You should probably also learn a little about vi.

MVC

Back in the realm of pure programming knowledge, it's extremely valuable to learn about MVC and related concepts. XPages sort of brought this to the Domino world, but it suffered from lack of wholehearted adoption and its legacy baggage made it very difficult to really go full MVC even if you wanted to.

The good news is that essentially every other active framework encourages or enforces this approach. When diving into one, it can be a bit jarring at first - there's a lot of "ugh, can't I just put the code here?" going on - but it falls into place very quickly.

So where to start?

Picking a place to start is a daunting prospect, given the ever-shifting sands of the modern development world.

One of my favorite tutorials for a web framework is Michael Hartl's Rails tutorial, which is kindly available for free online (with options to purchase other forms). It not only provides an up-to-date introduction to Rails development, but also covers a lot of the other ground mentioned above: the command line, Heroku, Git, and test-driven development.

In another direction, I recommend taking a look at Meteor, which is something of an all-in approach to JavaScript-based development. It does an interesting job blurring the lines between client and server and makes a very compelling case for "JavaScript everywhere".

Beyond that, really just learning any development toolchain will help. There's not too much need to sweat picking "the right one" at each step - any knowledge gained in one will help in others, either directly or by giving you some additional perspective. The most important thing, especially now, is learning something.

Change Bitterness and Accidents of History

  • Jun 17, 2016

It's pretty easy to see that change is in the air for Domino types. It's been taking a number of forms for a while now - the long delay since the release of 9.0.1 and associated aging of the tools and infrastructure have led to a series of forced adaptations for developers and administrators. Developers, for example, have had to keep light on their feet to adapt to new browsers and devices that the framework doesn't automatically support, as well as a shift toward manually including jQuery and other tools that have a bit more wind at their back than Dojo. Administrators, for their part, have had a series of heart attacks related to SSL and other security matters, usually involving a a lot of noise followed by (unfortunately, I feel) a good-enough patch from IBM.

That sort of thing isn't likely to get any smoother. In large part, that's entirely distinct from anything IBM does: the genie's out of the bottle when it comes to fast-moving platforms, and the best we can hope for is a sort of still-moving linga franca that can be sort-of-stable on the majority of targets. But then part of that is Domino: for all its virtues, it hasn't adapted for the modern world, no matter how much some of us would have liked it to. And that's had some negative side effects on us as a customer base, side effects that manifest as a gut-reaction rejection of the modern ways of doing things.

Take the SSL thing, one of my soap boxes: though the immediate problem could be summarized as "IBM should update their SSL stack", the larger issue it exposed was that our beloved monolith is mortally vulnerable to a single component falling behind. And, in fact, it made clear that we're spoiled by the approach: many of the reactions were basically that we shouldn't have to worry about things like reverse proxies or the general notion of distinct systems for web front-ends and the app server. And that initial rejection of the hassle has implications, limiting the average Domino installation's capacity to scale for load balancing or failover in a smooth way, things that come almost "for free" with a reverse-proxied setup common among almost all other app servers.

Developers have it worse: the "power user turned developer" history of Notes and the particular peculiarities of the platform* have left us desperately behind the baseline for modern development. The tooling has coddled us into preferring inline scripts and procedural programming to structured code organization, into viewing a thoroughly-staid language like Java as something outrageously complex, of only begrudgingly adopting SCM due to the platform-induced hassle, and of almost entirely ignoring automated testing. And similarly to the SSL discussion, the historic "one tool for every job" nature of Domino leads to natural pushback when faced with other platforms.

And I get why! And I feel it too. It's a real PITA to now always have to be on the lookout for some new point release of iOS or Android to break drop-down boxes or something, as opposed to years of deploying Notes apps that looked and worked identially across every version forever (pretty much). It's also a drag to run into situations where the problem is "easy" on Domino but more cumbersome elsewhere, like platforms that farm out their FT indexes to distinct servers, or don't include document/record-based security. That makes it very easy to become blinded to the tradeoffs, though. It may be nice that Domino is a one-stop-shop for so much, but it's a shop that requires Designer, that makes it very difficult to use third-party-dependency systems like Maven (even within Maven projects), that lags in DB features found elsewhere, that is only awkwardly accessible from other app-dev frameworks, that has an API that's a bit older than Windows Me, and that essentially never showed up for the modern development conversation.

Domino has always had a lot to recommend it, and XPages has carried us very far. And hey, this is enterprise software - even if there's never a major new version, there'll be paying work forever. It just may not be the kind of work you want to do, and it is almost definitely not truly healthy for the companies paying for it. So what I recommend is that you have a plan. The good news is that there are a great many next steps that build smoothly on existing Domino knowledge and, potentially, infrastructure. Certainly, I'm thoroughly biased in the direction of Darwino, but that's one of many. You could also do worse than learning a mature platform like Ruby on Rails (heck, you could run that on JRuby on Tomcat or WebSphere). Take some time to learn about reverse proxies and modern web-server setups. Basically: something. Just do it with an open mind, and don't balk at the first thing that's more complicated than the Domino equivalent. I think a bit of that will serve you very well.


* And, to be fair, of the overly-conservative nature of enterprise programming.

Other than, of course, being one of the progenitor NoSQL databases.

Code Safety and Pedantry

  • Jun 3, 2016

Lately, I've been musing a lot on the topic of code "correctness" - that is, beyond the normal case of wanting code to do what I intended, and further into the realm of sweating even extremely-miniscule details. A lot of this is due to my continued watching of the evolution of Apple's Swift language (I highly recommend following Erica Sadun's blog for this). Swift is very much in the camp of "make sure all your 'i's are dotted and 't's crossed" languages, as opposed to more fast-and-loose languages like JavaScript or Ruby.

I've gone back and forth on the overarching concepts from time to time. I've long been a big Ruby fan, and a lot of that is because of a general feeling that, if you let go of a lot of the "strict old aunt of a compiler" restrictions, you gain a tremendous amount of expressiveness and productivity with few real-world problems. On the other hand, being immersed in Java all the time has shifted my brain to appreciating the benefits of stronger compile-time checks (at least on paper). Overall, I'm more on the latter side than the former now, double-edged sword though it is. This is why I've been diving into things like aggressive null analysis in my code. Even when it seems like it's being a pedant, there are certain classes of bugs that it finds that I wouldn't even normally think of on the fly. For example, the null checker flags this as being a potential NPE:

if(this.foo != null) {
	this.foo.doSomething();
}

My first reaction upon seeing that was along the lines of "you're full of crap, Eclipse", but then I noticed the small path a bug could take to creep in: multithreading. If I'm in a situation where the object containing foo is used across threads, there's a possibility where Thread A would evaluate this.foo != null to true and start to step in to the block. Then, Thread B would get its turn on the processor and evaluate this.foo = null in another method. Thread A would then pick up and try to call doSomething() on the newly-minted null. So I've swallowed my pride and started writing safer code like:

SomeObject localFoo = this.foo;
if(localFoo != null) {
	localFoo.doSomething();
}

What I've always admired about Swift is that it takes these sorts of lessons to heart and adapts the syntax to suit. My interest in code-correctness pedantry in Java has led me to write out verbose abominations like this:

private final @NotNull String foo;

Three of the conceptual tokens there are purely to say things that are best practices to start with: I don't want this property accessible outside the class, I want to make sure it's assigned during construction and not change thereafter, and I want to ensure it's not null. The Swift variant is:

let foo: String

Same thing, half the typing. And, as a bonus, since nullability checking is built in to the language and not a by-convention thing like the null annotations in Java, I can be sure that the rules will be applied. That sort of thing is the dream! But, since Java is the best language for the work I'm doing for now, the important thing is that it at least suits, verbose or not. In a lot of languages, it gets much more difficult to have this sort of assurance.

So, hassle as it is, I suggest that other Domino developers, on their paths through Java, consider picking up the same habits. For every time you run into something like Java complaining that it can't convert a List<String> into a List<Object>, diving fully into null checks and immutability will save you a late-night crash report and angry user. As you develop more in Java, give it a try.

Darwino for Domino: Conceptual Overlap and Distinctions

  • Jun 1, 2016

I've talked a bit so far about how Darwino related to Domino from a development perspective, but I think it'd also be useful to delve into specifically which concerns the two platforms address, to see where they overlap and where they don't.

There are two main categories to cover, since Darwino inherits Domino's unusual trait of pouring over from "database" to "app-dev platform".

Database

As I covered a few posts ago, the two are similar at a conceptual level, both being replicating document databases with document-level access control. Aside from the difference between an NSF note's format and a JSON document, the main distinction is that Darwino doesn't cover the actual physical storage of data. Instead, it is based on top of existing SQL servers of various stripes (PostgreSQL and SQLite being the most common). A Darwino application creates a series of tables and uses them as the backing store for the conceptual document database.

This has a number of implications. The main one is that there isn't a "Darwino server" as such - instead, there are SQL databases and Darwino applications acting in tandem. In developing an application, this isn't generally a concern: the Darwino APIs are the same across each database, in the same sort of way that a Domino application doesn't care about the ODS version. However, being backed by a SQL server has some distinct advantages: the server can be administered and optimized using the same knowledge you would use for a "normal" SQL-backed app, and the ability of modern DBs to index JSON data opens up a world of possibilities (think NSFDB2, except good).

The flip side of this bleeds into the second category, as it means that a Darwino application consists of at least two parts: the SQL database and the application, which veers from Domino's "everything in one package" promise slightly.

Application

Things diverge most significantly (though at least as promisingly) when it comes to the application level. Domino has a few "official" ways to develop applications (Notes, legacy web, and XPages) and then hooks to sort of act like a Java EE server, albeit with some notable limitations. Darwino, on the other hand, exists as a sort of "glue layer" in between the database and the application: lower-level than XPages but higher-level than just a database driver.

Darwino provides a common platform for writing Java-based applications, with various services for managed beans, user directories, and so forth, written to work consistently across all of the platforms it targets. Again, this starts out similar to Domino, but diverges in the areas where Darwino takes advantage of other technologies.

At the low level, since Darwino's main requirement is "a Java runtime", it is able to run smoothly on various Java EE servers, on Android, on iOS, and on pretty much anything that provides a capable-enough Java environment (such as, say, Domino). It also, incidentally, means that it works great on Java 8.

At the high level, Darwino doesn't prescribe a specific UI framework, so the field is open to use any of the tremendous array of rapidly-developing Java frameworks on the web side and, as desired, native UI toolkits on mobile. There's a bit of an inherent bias towards REST+client JS applications, since then the same code entirely can be used on both web and mobile (as not every Java web tooklit works on the mobile mini web server as it is now), but that's not obligatory.

Overall

So the overall idea is that Darwino doesn't solve every problem that Domino does, but the problems it chooses to farm out are in the areas where that brings tremendous benefit. In each area where Darwino uses third-party support, it benefits from the tremendous advancements made in recent years, without requiring jumping through weird hoops to get modern techniques to work.

Speaking at Social Connections Toronto

  • May 30, 2016

By virtue of one of the original speakers having to cancel, I will be presenting at Social Connections in Toronto next week! Specifically, it will be on the 7th at 2:30, and the topic will be OpenNTF's new tooling and initiatives:

OpenNTF has long been a central hub for community and open-source development in the Notes/Domino world. In the past year, however, it has also expanded its capabilities with new tooling and a broadening scope to the larger IBM portfolio. This presentation will discuss OpenNTF’s history and its plans for the future, including a look at its newly-launched suite of source control, issue tracking, and continuous-delivery tools.

So, if you're attending Social Connections, drop on by the Toronto Room on Tuesday afternoon.

Old App Idea: App Manager

  • May 24, 2016

Years back, I took a small whack at an idea that had been percolating in my head: a "app manager" application that would assist with the XPage-specific portions of running a Domino server. It would cover a lot of ground that Administrator really doesn't touch, such as inspecting NSFs to see which contain XPage artifacts, highlighting potential problems with them, and assisting with app-design backup and deployment.

It never really got too far, since there are always a hundred other things to work on, but I always liked the idea. I decided to toss what code I had laid down up on GitHub:

https://github.com/jesse-gallagher/app-manager

I don't imagine I'll really have time to build on that unless I'm suddenly really struck with inspiration. But either way, the code is there for anyone interested in taking a look.

The Cleansing Flame of Null Analysis

  • May 21, 2016

Though most of my work lately has been on sprawling, platform-level stuff or other large existing codebases, part of it has involved a new small app. I decided to take this opportunity to dive more aggressively than previously into automated null analysis and other potential-bugs tools.

What I mean by "null analysis" is letting the IDE or compiler try to help you avoid NullPointerExceptions. Though there are plenty of other programming mistakes you could still make, these are among the most common, and so a little extra work up front to avoid them should pay dividends. Eclipse has some handy options in its Java → Compiler → Errors/Warnings preferences to assist with this:

The first option will pick up on some pretty basic instances, such as:

Object foo = null;
System.out.println(foo.hashCode());

Since this is clearly going to always cause an NPE, Eclipse is able to point this out as an error. The next level gets a little more nebulous: "potential" null pointer access. This crops up when Eclipse can't reliably determine whether a value will be null, either because there is no way to know at compile time (say, database access) or because the compiler's tooling is too limited. Here's a contrived example:

Object foo = Math.random() > 0.5 ? new Object() : null;
System.out.println(foo.hashCode());

This situation is clearly untenable, but there are other situations where you as a programmer can be very confident that the value will not be null (say, if you swap out the > 0.5 for >= 0.0), but the compiler doesn't know that. That's why it often makes sense to leave that as a warning instead of an error.

That's all stuff I've done before, but now I've decided to dive into annotation-based null analysis as well. Unfortunately, in stock Java, this is something of a hot mess (that list even leaves out Eclipse's home-grown version). Since Java didn't grow up with this sort of capability, it's been shoehorned in by various parties over the years. There are other tools to assist you in Java 8, but, unfortunately, I can only target 7 as the highest. For now, I've settled on the "sort-of standard" javax.validation.constraints package. It wasn't really intended for this specific purpose, but it's flexible enough to suit and can be used in Eclipse and FindBugs (though I have my reservations about the choice).

In Eclipse, this type of analysis can be enabled by checking "Enable annotation-based null analysis" below the other options and, unless you're using Eclipse's known annotations, adjusting the "Configure" options next to "Use default annotations for null specifications":

In any event, regardless of the choice of tooling, the "this shouldn't be null" annotations work the same way: you use them to decorate things that you either require not be null when provided to you (method parameters) or you promise to not be null when providing to others (method return values). For example:

public @NotNull Object doSomething(@NotNull Object otherObject) {
	return otherObject.toString();
}

This highlights three things, two good and one bad:

  • Good: The @NotNull in the method parameter means that, as long as the calling code is also checked for null use, the method can be confident that there won't be a NullPointerException when calling a method on otherObject.
  • Good: The @NotNull on the return value means that other code calling this method can be confident that they will not get a null value from it, and so can skip extra null checks.
  • Bad: Eclipse flags otherObject.toString() as a potential problem because it doesn't know for sure that Object#toString doesn't return null, because it has no nullability annotations. As programmers (or as a compiled-code analysis tool), we can be fairly confident that it will be non-null because any object returning null for that is essentially broken on its own.

That last one is a common problem when adopting annotation-based null analysis, at least in Eclipse (I hear it may be better in IntelliJ): its logic doesn't go very deep. If everything is gussied up with these annotations, you're clear - but as soon as you step outside of the project you're working on, you have to add in likely-unnecessary checks. Fortunately, these checks don't realistically hurt (a null check at runtime in a normal app is negligible performance-wise), but they can grate to have to add in.

Glutton for punishment that I am, I decided to go a step further and enable FindBugs processing as an integral step of my build. Though FindBugs can be very picky about the types of things it complains about, it is blessedly more thorough in its analysis than Eclipse, so you generally end up conceding that it is correct when it yells at you. Since the project is Maven-based, I added the check in the project's pom file:

<plugin>
	<groupId>org.codehaus.mojo</groupId>
	<artifactId>findbugs-maven-plugin</artifactId>
	<version>3.0.3</version>
	<configuration>
		<includeTests>true</includeTests>
	</configuration>
	<executions>
		<execution>
			<phase>compile</phase>
			<goals>
				<goal>check</goal>
			</goals>
		</execution>
		<execution>
			<id>findbugs-test-compile</id>
			<phase>test-compile</phase>
			<goals>
				<goal>check</goal>
			</goals>
		</execution>
	</executions>
</plugin>

For most uses, that's all that's required. Now, when the project is compiled, FindBugs will give it a once-over and halt the build if it finds anything it doesn't like. This can be tweaked a great deal - for example, changing the checks to run or the severity of the problem needed to fail the build - but the defaults will likely suit.

Adding these extra checks involves a lot of plusses and minuses. The big minus is that you may end up spending a lot of time "fixing" bugs that don't really exist, time that you could instead spend actually writing your application (and writing new bugs that the tools won't find anyway). There's really nothing to be gained by carefully explaining to Eclipse for the hundredth time that toString always returns non-null.

Still, particularly when tested out in a small, low-surface-area app, this can be a good practice to learn and refine. Eventually, a move to Java 8 will help this more, and it certainly doesn't hurt to add in nullability annotations in the mean time. Overall, I think having the tooling help you avoid a whole suite of common "brain fart" bugs like this is worthwhile.

Quick, Short-Notice Announcement: Delaware Valley Soc-Biz UG Meetup

  • May 17, 2016

Granted, this is short notice, but for anyone in the southeast-PA area, there's a meetup at IBM's offices this Thursday (the 19th) from 11 to 12:30. I'll be there, giving a presentation about OpenNTF and some of the ways that I've used the projects we work on there on my customer projects. Better still, there's lunch! The signup form is over on Greenhouse here:

https://greenhouse.lotus.com/forms/landing/org/app/80846239-6f7a-4483-8ace-9e5e02b0a661/launch/index.html?form=F_Form1

If you're able to make it, great! Otherwise, I imagine there should be more of this sort of thing in the future.

Darwino for Domino: Domino-side Configuration

  • May 16, 2016

In my last post, I mentioned that a big part of the job of the Darwino-Domino replicator is converting the data one way or another to better suit the likely programming model Darwino-side or to clean up old data. The way this is done is via a configuration database on the Domino side (an XPages application), which allows you to specify Database Adapters that configure the translation. While it is possible to write these in Java, the primary way is to use a Groovy-based DSL script.

The simplest form of this script may be one line just defining where the NSF is:

nsfName "foo.nsf"

With that, the replicator will open foo.nsf and attempt to replicate all documents with "best fit" translations of each field it comes across. Things can get a little more complex, though:

form("SomeForm") {
	excludeField "foo"
	excludeFields "IgnoreMe"
	
	field "foo_(.*)", nameRegex: true
	arrayField "bar_(.*)", delimiter: "\$", zeroBased: true, initialIndexed: false, prefix: false, compact: true,
		userDataFormatName: "ODAChunk", nameRegex: true
	field ~"impliedfoo_(.*)"
}

form("ArrayTest") {
	arrayField "FirstName", type:TEXT, delimiter: "_"
	arrayField "LastName", type:TEXT, delimiter: "\$", zeroBased: true, initialIndexed: false, prefix: true, compact: true
	
	// Similar to the ODA style
	arrayField "UserData", type:USERDATA, delimiter: "\$", zeroBased: true, initialIndexed: false, prefix: false, compact: true,
		userDataFormatName: "ODAChunk",
		toDarwino: { value ->
			// The value will be an array of byte arrays, which should be strings
			println "testrep: called toDarwino!"
			value.collect { bytes ->
				new String((byte[])bytes, "UTF-8");
			}
		},
		toDomino: { value ->
			// We should provide an array of byte arrays
			println "testrep: called toDomino!"
			println "value is ${value}"
			def result = value.collect { string ->
				string.getBytes("UTF-8")
			}
			println "result is ${result}"
			return result
		}
}

form("RestrictedFields") {
	restrictToDefinedFields true
	field "KnownField"
	field "KnownField2"
}

The specifics of what's going on in this example (pulled from unit test data) aren't too important, but it demonstrates the customizability that an in-language DSL brings. Since the code is executed in Groovy, it has access to the full Java runtime (including, if you deliver it in a plugin, your own custom classes and dependencies), as well as Groovy's nice abilities like closures. In fact, a great many properties, like the converters above, can be specified as closures and executed in the context of each document as it's replicating, allowing for pretty fine-grained translation.

And personally, adapting Groovy into the process was an interesting exercise. Since Groovy was designed explicitly as a "scripting" variant of Java, the process of working it in to an existing Java code base is very smooth, and there aren't too many gotchas. I wrote some Java classes that provide the context for the root and individual "form" blocks, wired up the interpreter, and then they call each other seamlessly. Other languages could probably suit the job well too - JRuby, Rhino, etc. - but Groovy is mature, purpose-built, and largely syntax-compatible with Java itself, making it a very comfortable fit.

Darwino for Domino: Replication and Data Format

  • May 11, 2016

One of the key points of interest in Darwino for Domino developers is its two-way replication. Darwino's replication system was built in such a way that, in addition to its own internal needs, you can also write a replicator to connect to an entirely-unrelated system, as long as that replicator translates the foreign data to and from JSON documents. Domino is a perfect case for this, since the data model is already very similar, and its replicator ships with Darwino and has been a focus of my attention for a while.

Behind the scenes, this replication takes advantage of a lot of the same kinds of things that Domino's replication always has: UNIDs, sequence IDs, original-vs.-in-file modification/creation, and deletion stubs. These details are transparent to the developer: the Darwino adapter knows how to fetch the appropriate data from the NSF and to convince Domino that the Darwino DB is (almost) like a remote Domino server, at least as far as the stored data is concerned.

The primary differences show up in the way data is formatted for storage. Darwino uses JSON for its document format, which has a couple key advantages and disadvantages compared to NSF's "bag of items" approach. The best approach is probably to provide an example and highlight the pertinent differences. Say you have a Domino document that (conceptually) looks like this:

FirstName
	Type: TEXT
	Value: Foo
LastName
	Type: TEXT
	Value: Fooson
Username
	Type: TEXT
	Flags: NAMES READWRITERS
	Value: CN=Joe Schmoe/O=SomeOrg
Birthday
	Type: TIME
	Value: 1970/2/1
Vacations
	Type: TIME_RANGE
	Value: 2016/1/1-2016/1/5, 2016/3/3-2016/3/5
IsAdmin
	Type: TEXT
	Value: Y

On the Darwino side, that would potentially look like this:

{
	"_writers": {
		"username": ["cn=Joe Schmoe,o=SomeOrg"]
	}
	"firstname": "Foo",
	"lastname": "Fooson",
	"birthday": "1970-02-01",
	"vacations": [
		"2016-01-01/2016-01-05",
		"2016-03-03/2016-03-05"
	]
	"admin": true
}

There are certainly a few things to take note of here. First and foremost is the structure of the authors field. Because JSON doesn't have field metadata, the way Darwino does its readers/writers security is by using specially-named and -structured properties within the JSON, and so the converter moves all readers and writers fields into that. This has internal-implementation reasons, but I think it's also conceptually preferable to, say, having a multi-level object within each field to declare its flags separate from the value. The name also happens to be stored in LDAP style, because Darwino is more at home with standard LDAP conventions for that sort of thing.

Another thing to note is the format of the date fields. Since JSON doesn't have a real date/time type of its own, these values are converted according to ISO 8601 and stored as strings. That means that your Darwino application will need to know that those string values represent dates, but they're reasonable to deal with.

The multi-value date field leads to another important aspect: arrays. In Domino, most items are conceptually presented as arrays, regardless of whether they contain single or multiple values, leading to code that requires either explicitly asking for the first element or jumping through hoops to deal with single or multiple values. Since that's a drag to worry about with JSON, the default behavior for single-value items transferred to Darwino is to store them as "bare" values. When configuring the translation (which will be fodder for a future post), you are able to specify that you want a field to be always stored as an array, which will allow the Darwino-side code to be simpler.

The last field shows off an outright advantage of the JSON format: boolean storage. When defining the conversion, you can specify a field as boolean and provide what the true/false values will be, and they will be sent over to JSON as true and false explicitly. That's not a night-and-day change, but it is a nice help.

Finally, there's the matter of rich text, which is unsurprisingly a nontrivial problem. This is handled in an XPage-alike way: MIME rich text is transferred with only minor adjustments, while Composite Data is converted to HTML and cleaned up a bit before transfer. Darwino supports the concept of attachments natively, and so Domino attachments are brought over with a naming prefix to match the field they're attached to, plus a delimiter to indicate whether they're normal attachments or embedded images. The way it is presented on the user end is dependent on the application, but Darwino has some routines to translate storage-safe inline image refs to app-relative URLs.

Later, I will go into the process of how the Domino-Darwino adapters are set up. The short of it is that you create scripts that can run the gamut from just telling the server where to find the NSF to customizing the translation of each field encountered. This allows you to either transfer the data back and forth in the default "best approximation" approach or use the opportunity to enforce a bit of a schema on old data.

An Overview of Darwino for Domino Types

  • Apr 14, 2016

So, Darwino! I've mentioned it quite a few times on Twitter and, particularly, in person, but I think it's high time I write some proper blog posts about it.

To start with, I'll cover what Darwino is. The short version is it's a Java-based development framework with a replicating document database. The interesting aspects go beyond that, though:

  • In addition to Java web servers, it targets mobile devices, both Android and, through RoboVM, iOS. Those devices store their own replicas of the databases for offline work in the same conceptual way as Notes, but with native (or hybrid web, if you're so inclined) mobile user interfaces.
  • The document database sits on top of SQL servers. Many modern SQL servers have native support for JSON data, and Darwino takes advantage of this to get document-DB flexibility with SQL features.
  • Business logic is shared between platforms. Because Java acts as a common language between each platform, and the document DB works the same way locally and remotely, the core business logic of the app can be identical across each targetted platform, with only the UI changing between them.
  • Along those lines, Darwino isn't prescriptive with the UI: it's not a front-end framework itself, instead providing the basis for using other front-end tools, such as Ionic, JSF, and Vaadin for web/hybrid UIs and the native OS toolkits on mobile.
  • The Darwino syncing protocol is designed to be adaptable to other services. This is immediately notable for Domino developers, but can also be (and has been, in some cases) adapted for arbitrary other back ends, like Connections social data or other databases.

How does this relate to Domino/XPages development? That depends on your desires, really.

In some ways, it doesn't. Darwino is its own platform, running on Java web servers like WebSphere and Tomcat, using independent SQL servers like DB2 and PostgreSQL. Darwino's replication between the server and mobile devices is similar to Domino's, but is its own thing. Similarly, the document model, though conceptually similar to Domino (including enhanced reader and author fields), is not NSF.

However, there are a number of reasons why it's of interest to a Domino developer, and the most immediate of those is its ability to do two-way replication with Domino databases. I'm a little biased on this point because of how much time I've spent working on it, but this replication is capable of some nifty tricks to make it capable and adaptable, including transformation of the data, two-way maintenance of document time stamps, and so forth. With this syncing, it makes it very practical to extend your existing Domino app - be it a classic-style Notes/web app or an XPages one - with a Darwino-side UI that uses the same data, synced down to mobile devices for offline access. And this doesn't require migration; since the changes replicate back, the app can remain chugging away unchanged on the Domino side if desired. This also can be tremendously useful for reporting, by syncing the data over to a full SQL database that can be viewed and queried by normal tools.

And, really, it's also of interest to Domino developers personally by virtue of being a platform that has learned a lot of valuable lessons from Domino and extended them in new ways. Those years of accumulated document-database knowledge will carry over nicely, with some extra benefits if you're SQL-familiar too. Any Java knowledge will come in handy immediately, as Darwino is thoroughly Java-based on all target platforms. And, thanks to its pedigree, a lot of the platform support concepts are similar to aspects of XPages (the good parts). In general, the more XPages development you've done, the more it will benefit you (especially if you want to use JSF for the UI!). You could also, with some servlet-implementation limitations, run Darwino apps on Domino via OSGi, and I've been putting some side work into accessing Darwino databases from XPages directly - Darwino could make a solid basis for Domino-run apps.

So this turned into a bit of a sales pitch, but there's no getting around it - I find this thoroughly compelling and exciting. As I have time, I plan to expand on Darwino's various capabilities (along with the other various blog series I still plan to get to). For now, you can register for and download the Community Edition, read the documentation there, and/or track me down with any questions.