Showing posts for tag "domino"

Java With Domino After XPages

Mar 14, 2019 3:10 PM

IBM and HCL held a webcast today to detail some plans for Notes/Domino V11. There were some interesting tidbits elaborating on things like the pub/sub support, and it'll be worth tracking down a recording of the event when it's available.

What's important for this series, though, is that this event served as the long-promised "roadmap" announcement for XPages. The roadmap is, in effect, option three: HCL plans to look into ways to reuse some existing XPages code, but in general you should be aiming to write your UIs in something else, either consuming REST services from an XPages container or accessing Domino data via another route (like the domino-db Node.js module and hypothetical Java gRPC client).

So we know the end of the path: not XPages. However, it's not like we're all just going to throw away our existing apps, so there's work to do determining how we're going to get there. The options remain pretty much what they were after CollabSphere last year, albeit now with the doubt removed. The first two options - returning to LotusScript or going to Node - have their advantages and disadvantages, and you could make a reasonable case for either. Personally, I'm not interested in going down those roads, though, and I think it's better for any app of reasonable complexity to dive into Java. Other members of the community and I have developed tools over the years to make it easier, and now's the time to take some of these steps if you haven't already.

Do Not Use Server JavaScript

Sever JavaScript was always something of a trap for app architecture. There's nothing inherently wrong with having a scripting language on your UI pages, and it certainly helped bridge some gaps, but the way it and Designer intertwined encouraged developers to create non-portable messes. If you're still writing SSJS, stop immediately.

Learn Proper Java

Java has been around for a long time, and the way to right "good" Java code has changed over time and varies greatly by your environment. Some aspects, though, apply generally, and it's useful to stay up-to-date on current practices. I don't know a better resource for this than Effective Java, which has been updated for Java 7-9 since I last read it.

Speaking of which, you should learn about Java 8 streams and lambdas - they're great. Julian Robichaux did a presentation on this topic back at Connect 2017, and the slide deck is very elucidating.

Adopt Standard Java Technologies

Last year, I created a project to bring some modern JEE technologies to XPages. These are some of the same technologies I've been talking about in my "XPages to Java EE" series and, while that project can't bring the full JEE development experience to XPages, using those tools will help you write code that, in some cases, could be directly dropped into a Java EE app with no modifications at all. There's a big asterisk when it comes to actually accessing Domino data, but that's a solvable problem as well (with some more development).

In particular, you should start writing JAX-RS services. Not only is JAX-RS an excellent and very-capable spec, but REST services are portable to absolutely any front end.

Adopt Automated Builds

Maven has been something of a bugaboo for XPages developers for a while, but doesn't have to be. Node development (server- or client-side) revolves around npm and various build plugins, and Maven is much the same thing. One of the biggest improvements I've made lately to all of my active XPages apps is to wrap the on-disk project for them inside a Maven artifact, using the NSF ODP Tooling. That project allows you to automatically build your NSFs alongside other parts of the project (such as OSGi plugins) without having Designer involved.

Check the example project in that repo, and stay tuned for a 2.0 release (probably) imminently.

Learn Other Toolkits

If you're just starting the process of figuring out what to do after XPages, it doesn't particularly matter which other toolkit you learn, as long as it's reasonably modern. If you take some time to learn how to make, say, a React app but end up going with something else down the line, the lessons you learn will apply very closely. A particularly-comfortable option could be to learn JSF, which has a common ancestry with XPages but has up-to-date capabilities.

Whatever it is, though, just learn some other toolkit.

Follow Channels and Accounts for Other Tech

Over the last couple months, I've started following a lot of Jakarta-related blogs and Twitter luminaries. This applies elsewhere - even if you're not using other toolkits yet, it's very helpful to start immersing yourself in the news and culture.

Don't Stay Still

The primary thing to take to heart is the importance of doing something. Unless you're planning to change careers or retire in the short term, you'll have to make one decision or another. XPages is not going to get meaningfully better, and even existing apps will get worse with time as browsers and technology change.

Other environments, though, are already leagues ahead and are constantly improving. Dive in; the water's fine!

Domino 10 for Developers

Oct 9, 2018 1:17 PM

Tags: domino

So Domino 10 is upon us, marking the first time in a good while that Domino has had an honest-to-goodness version bump.

More than anything, I think V10 is about that sort of mark. Its primary role in the world is to state "Domino isn't dead" - not exactly coming from a position of strength for the platform, but it's the critical message that HCL has to sell if they're going to be viewed as anything but coroners.

Still, in addition to merely existing, V10 brings some changes that will help developers, particularly those - sadly - maintaining large legacy applications.

DQL

The addition that will have the largest immediate impact on developing codebases is, I think, DQL. I went into a bit of detail on this before and I think that that post is largely accurate, but the general gist of it is that DQL can be thought of as "database.search(...) but good", bringing practical arbitrary queries of non-FT data to Domino.

In its current form, it feels like a long-back-burnered passion project that's implemented in an effective way, bringing some of the benefits of arbitrary queries in SQL and new-era NoSQL databases without having to rewrite NIF or NSF storage.

UpdateAs Karsten Lehmann kindly pointed out, DQL is slated for addition to the LS/Java classes in 10.0.1 at an as-yet-unspecified time.

HTTP Methods in LotusScript

I just let out a heavy sigh after writing that header, but I get why they added these. A lot of Domino developers never left the desiccated-but-comforting embrace of LotusScript or are employed primarily to maintain Notes client apps, where using Java is possible but involves jumping over hurdles.

Network operations have been possible for a long time via OLE (on Windows) or LS2J, and I'm sure I'm not the only one who's had a "Network" LS script library sitting around for over a decade, but having baked-in methods is preferable. Moreover, neither of those mechanisms would work on iOS without a lot of additional work.

This is being billed as enabling all sorts of integrations, which I suppose is strictly true in that it's a bit easier to call HTTP methods in old code now. In practice, I think it will be mostly helpful for the little one-off situations where you have to call some web service to integrate with a product tracking app or the like.

iPad Notes Client

This definitely seems like another back-burner project that was brought to the fore in the HCL transition. They had iOS references in the Mac 64-bit C SDK years ago, and it only makes sense, since the existence of the Mac port at all meant the job was (sort of) half done. It's not out today, but it's logically tied to V10, and they've been expanding a beta over the summer.

Like the additions to LotusScript, it makes sense. I can't imagine that running existing Notes apps on an iPad will be a good experience, but it should be a cheap one, and it'll probably be good enough for at least some cases. They've intimated that there will be affordances in app design to improve the experience specifically for this client, though I don't envy the engineers who have to go in and implement those.

Node.js Support

Dubbed the "App Dev Pack", Node support will be coming in an Upgrade-Pack-like additional download, in the form of a Domino server addon to add a gRPC server combined with a domino-db Node module that I gather is designed to be familiar for Node+MongoDB stack users.

When this intention was first announced, I think that a lot of Domino developers figured it would be like XPages: a new design element or two added to the NSF, plus another runtime crammed into Domino's aging HTTP stack. The other big potential option was essentially a codification of the ExtLib DAS REST services into a wrapper package to be used in standalone Node apps.

The App Dev Pack is more the latter than the former, but the use of gRPC should make it more performant and flexible than just wrapping the existing HTTP services. I'll be curious to see how this shakes out in practice. XPages has been with us for a decade, but it still only captured a slice of the Domino development market, and it carried the advantage of being bundled right into the stack. Node is a very different beast, entirely unlike traditional Domino development, and I'm not sure how many existing Domino developers will make the transition. Ostensibly, one of the main benefits is to also attract new blood, which - well, maybe.

Having this is much better than not, and the notion of having a new RPC connection that doesn't have the local runtime requirements of NRPC is tantalizing.

Overall

Overall, this release definitely feels like a very pragmatic release. Just by virtue of its existence, it covers the base of "Domino isn't dead" in a way that's much better than the older mealy-mouthed messaging of "well, we don't have specific plans to cancel it". Additionally, though, the fact that most of the developer-facing improvements are for "old world" design elements is an acknowledgement that XPages didn't capture the Domino development world (and, probably, that HCL didn't hire the XPages team). The prospect of the community crawling back into the LotusScript cradle isn't great, but there's no avoiding the fact that there are a great many developers who never had a reason to do anything different. Not many cost-cutting IT departments let their developers re-learn their entire skillset when other departments are just asking for a new button on a form.

In an alternate universe, this would have made for a fine "Domino 9.5" release, but the wringer that the 9.0.1 era put us through demanded a full major version bump. I'll be curious to see how Domino 11 and so on shape up. If the "not dead" push works and it turns Domino's fortunes in the market around at all, it would give HCL room to turn it into a real platform again. That's a big "if", since it's a lot easier to get existing Domino developers excited than it is to get IT purchasers to sign the licensing checks, but time will tell.

DGQF and DQL as I Understand Them

Jul 26, 2018 12:27 PM

Tags: domino dgqf

At CollabSphere this year, the big information coming from HCL was detail about the Domino General Query Facility (DGQF) and its associated language, Domino Query Language (DQL). They originally announced this a few weeks ago, but it was good to have had some time to let the dust settle and to see the specifics.

Because it was discussed alongside the domino-db Node.js package and because it's one of the first real new ways we'll interact with data in a Domino DB in a while, it's a bit difficult to identify just what it is and what it is not. Here's how I understood it:

What DGQF Is

DGQF is, at least conceptually, a "meta" layer on top of the existing NIF indexing facility. It doesn't provide a core change to the actual storage of documents, but instead treats existing view indexes as (roughly) analagous to both SQL table indexes and SQL views. It trawls through the design elements of a database to analyze their selection formulae and columns to use applicable ones as implicit indexes and also to allow access to arbtirary collections within queries.

Implicit Indexes

Other than the design collection and the "optimize document table" option in a DB, an NSF doesn't really have much in the way of indexing note contents by default. So, if you have a query asking for all documents where FirstName is Bob, a program has no choice but to look through every document for that key/value match. If, however, you create a view that has a column showing the FirstName field, you now have a much-faster index you can use. It's this sort of view that the DGQF picks up on implicitly, using them to accelerate queries: views showing all documents with either a default sort or "click to sort" column showing explicitly a field (and not a formula).

Access to Arbitrary Collection Data

For those qualifying views plus others, you can reference a view by name or alias to compare to a column value by programmatic name (often either the field name for simple columns or something like $4 by default for formulas).

"In" clauses

Additionally, you can use view (and folder, I think) names to refine queries for documents that are in one or more of these collections, equivalent to an "in" subquery or view reference in SQL

What DQL Is

In short, DQL is the human-readable query language used to access DGQF. It's reasonably SQL-like (though it is not SQL) and tends to look like FirstName='Bob' and in all ('Managers', 'Active Users'). This is the language you will use, and so "DGQF" and "DQL" will generally refer to the same thing in practice.

In practice, this is implemented as a new method on the Database class in each high-level language supported by Domino, plus a Node-styled variant in domino-db.

What DGQF and DQL Are Not

Since DGQF sits on top of NIF (and probably the FT index eventually), it's not a core change to data storage. Eventually, the same abilities and limits of Domino remain as they are with respect to this.

Additionally, DQL is, I believe, a query language only: it does not provide a mechanism for creating, modifying, or deleting existing documents. Instead, it is essentially a super-powered and much-smarter version of database.search(…): you can use it to find documents and the processing of them is up to your program.

That last point was a bit muddied by its pairing with the domino-db Node.js package: the Node.js package provides bulk operations that are paired with DQL queries, but that is a function of that library specifically, not DQL or DGQF.

Why It's Cool

Though it's not a reworking of the core NSF, what DGQF does do is abstract away a lot of the manual looping and lookups that we've always had to do, and it allows the system to optimize and do things more efficiently than when written out procedurally. So, while there's theoretically nothing that DGQF does that we couldn't do before, it allows us to do those things with far, far less code and with automatic optimization.

This brings Domino something that SQL servers have enjoyed for a long time. With a SQL statement, you can analyze the trouble spots of a slow-running query and add indexes to improve the speed, with the tooling helping to explain what's going on. DGQF+DQL brings this along for the ride: when you execute a DQL query, you have the option to dump out this "explain" output to see what specifically the facility did, which views it used, and how long each step took. So, if you have a long-running query, you can look to see if you can add an "index" view to automatically speed it up without having to change your code. And, since the language is an abstraction over the task of querying and not the sort of "burned in" process of a normal getNextDocument loop, it can be optimized and short-circuited by the underlying system without the developer having to know the decades of built-up knowledge of how to efficiently search a DB.

All in all, this is a very welcome addition to the server, and it certainly should improve a lot of common tasks.

A (Java-Centric) Domino Wish List

Jul 12, 2018 12:04 PM

Tags: domino java

Seeing the information come out of this week's HCL "Golden Ticket" event has got me thinking about some of my wish-list items for Domino development, mostly in the form of enhancements for existing capabilities and entirely around Java (since that's what I do).

Quality of Life

Javadoc

For some reason, the lotus.domino classes ship without Javadoc or even variable-name information, leading to this trainwreck:

Designer has its built-in help, which is also on the web, but that's quite a few steps down. This is table stakes for a Java API and always has been.

Updated p2 Repository

Back in 2014, the XPages team uploaded a clean p2 repository of the XPages artifacts to OpenNTF, corresponding with the 9.0.1 release. This repository saves a ton of hassle when building Tycho-based projects or just setting up an Eclipse workspace. However, it's quite long the tooth, as there have been several Notes.jar additions not included in there, and, in FP10, a significant upgrade to the undergirding OSGi framework.

I ended up writing a script to generated an updated version, but I don't have the legal ability to publish the results anywhere for easy consumption, meaning it has to be done manually and configured for each build environment. It would be a great convenience if there was an official package (ideally including the Designer plugins as well) and, even better, hosted on OpenNTF so that we could reference it by URL as we do for Eclipse releases (and require users to accept a license first).

Mavenized Repository

The p2 repository is good for Tycho-based projects, but, especially when targetting Domino is only one part of a project, it can be much more convenient to use "normal" Maven projects with maven-bundle-plugin. However, those projects can't use p2 repositories as such. For Darwino's needs, I ended up writing a tool in the (available-for-free) Darwino Studio plugins to Maven-ize a Domino p2 repository, but that hits the same snag as above of requiring manual setup in each instance.

This is another case where my preference would be on the OpenNTF Maven repository (plus Javadoc Jars, naturally).

Extension Library Source

The latest Extension Library release on OpenNTF is from FP9, while the latest on GitHub is from the FP7 era. FP10 shipped with a newer version of indeterminate nature. It'd be good to have this on both of those sites and, like with Javadoc, have source bundles shipped with the product in a way that is picked up automatically by Designer and Eclipse.

Source Bundles for Third-Party Components

The source for the undergirding Equinox stack is available, but it would be best to have, as an adjunct to the updated p2 repositories, the source bundles for the actual versions used so that we don't have to cobble together a platform from Eclipse's repositories.

Open-Source the Rest of the Stack

Having XPages, the Expeditor husk, and the other miscellaneous doodads that make up the proprietary layer as open source with an Apache-compatible license would cover a lot of the above and also be of tremendous use for XPages and non-XPages apps alike that run on or with Domino. I have a hard time imagining that it would lead to a lot of community-driven improvement, but it may do some (I'd have a few words to share with the file-download control, for example), and even just as a static release would be a significant boon.

Domino Connectivity in Eclipse

An idea I've been toying with lately is to make an Eclipse plugin that allows you to add Domino servers to the "Servers" view and control them to some extent. The basics would be to start/stop/restart HTTP, but the stretch goals would be to open a console view, get a list of running modules, integrate with the existing "load bundles from PDE" support, and, ideally, an outright "Run on Server" command for OSGi bundles and NSFs. However, I have so much on my plate that I'm not sure that I'll get to this any time soon unless I get a real itch some weekend.

Longevity

Refreshed JVMs

Feature Pack 8 brought Java 8, a vital step forward. However, since then, Oracle moved to a faster release cycle for Java and the JRE is now at version 10. Domino uses IBM's JVM variant, J9, which they recently moved to the Eclipse foundation as OpenJ9, where it has... sort of been keeping up, I think?

In any event, this increased pace of change has meant that the Java 8 honeymoon is over, and Domino development again requires special consideration when using current tools. I have no idea how complex the integration between Domino's tasks and the underlying JVM is, but my ideal would be to have constant or near-constant parity.

Servlet API 4.x

After the JRE version, the most important foundational element of a Java web app is the servlet API release. The current version is 4.0.1, while Domino supports 2.5 (or 2.4, maybe?). The good news is that the Java/Jakarta EE world seems to be used to lagging versions here, and 2.5 is a minimum version for a lot of current tech in much the same way that Java 6 was until somewhat recently, but there has been quite a bit added in recent years.

Presumably, a reason for the lag is the implied requirements of newer versions, such as WebSockets and HTTP/2 support, that would require heavy modifications to the core Domino HTTP code. Honestly, the more practical route is almost definitely to just use a different JEE server paired with CrossWorlds, some Java wrapper for the GRPC stuff HCL has been talking about, or (best of all) a Darwino app replicating with Domino, but still. WebSphere Liberty is actually really nice, by the way.

Refreshed Equinox

Like with the underlying JVM, the Feature Pack 10 update to a Neon-based OSGi/Equinox framework was a critical shot in the arm for the platform, but it is now also two major versions behind. This is a little less critical, since Equinox brings a bit less to the table for our needs and Neon is "new enough" for now, particularly on the server side, but it'd still be proper to keep pace.

Odds and Ends

Non-OSGi JEE Support

The Equinox framework that Domino uses is quite capable, but there's no pretending that OSGi-targetted development has its share of headaches. Most Java apps just target plain-old .war files and don't impose any particular requirements on the build process. Java development is a much more pleasant experience when you can just toss in any Maven dependency and not have to think about building a target platform for Eclipse or jumping through bundle-resolution hoops. I really like OSGi in theory, but I can't pretend that non-OSGi development isn't much smoother.

Domino technically supports "regular" servlets currently, but, uh, here's a snippet from the current documentation on that:

Hrm.

Full Extension Manager Support for Java

JAVADDIN/DOTS added a lot of EM hooks, but it doesn't cover the full suite of capabilities that a C addin can provide, such as authentication handling. Having this be fully accessible from Java would be useful even when treating Domino just as a data store and not as an app server.

 


 

I'm sure I could come up with more, but that's probably good for now. All easy, right?

First Steps to Code Coverage Analysis in Domino Plugins

Nov 9, 2017 8:53 AM

Tags: maven domino java

I'm always interested in getting the computer to tell me how to tell it what to do more successfully, and, to further that pursuit, I've started taking an interest in code coverage.

If you're not familiar with the term, "code coverage" refers to reporting on which lines of code were actually executed during runtime, most commonly in association with unit tests. Eclipse (and presumably other IDEs) has support for this, and I've decided to give it a shot.

Since I'm starting this out in the context of Domino plugins, there are more wrinkles than in most tutorials. Namely, the test suites I've written run exclusively through Maven instead of the Eclipse UI due to all the Notes environment setup, so I can't just use the normal UI tools to gather the data. Fortunately, Eclipse's EclEmma will work just fine with the output from a Maven project, as long as you configure it properly. I looked around for a while to find the right combination of tools to use, but it ended up being fairly simple to configure basic output that can be consumed in Eclipse's Coverage view.

There are two main additions. First, add the jacoco-maven-plugin to your root project's project.build.plugins block:

<plugin>
	<groupId>org.jacoco</groupId>
	<artifactId>jacoco-maven-plugin</artifactId>
	<version>0.7.8</version>
	<executions>
		<execution>
			<goals>
				<goal>prepare-agent</goal>
			</goals>
		</execution>
	</executions>
</plugin>

In normal cases, that would suffice. However, since the test configuration I have for Notes overrides the argLine property of the Tycho test runner, there's another step - add the tycho.testArgLine property manually into those blocks, such as in the Windows profile:

<profile>
	<activation>
		<os>
			<family>Windows</family>
		</os>
		<property>
			<name>notes-program</name>
		</property>
	</activation>

	<build>
		<plugins>
			<plugin>
				<groupId>org.eclipse.tycho</groupId>
				<artifactId>tycho-surefire-plugin</artifactId>
				<version>${tycho-version}</version>
 
				<configuration>
					<skip>false</skip>
 
					<argLine>${tycho.testArgLine} -Dfile.encoding=UTF-8 -Djava.library.path="${notes-program}"</argLine>
					<environmentVariables>
						<PATH>${notes-program}${path.separator}${env.PATH}</PATH>
					</environmentVariables>
				</configuration>
			</plugin>
		</plugins>
	</build>
</profile>

Once that's configured, running the test suite via Maven will create a new file in the target folder of the test plugin: jacoco.exec. This file can then be consumed in Eclipse by opening the "Coverage" view:

Eclipse's Show View window

In that view, right click and choose "Import Session..." and point to the data file. Click "Next" and check the projects+source folders from your workspace you're interested in analyzing. When you click "Finish", it'll do two things. First, it'll fill the Coverage view with statistics from your run:

Code Coverage stats

(We have a lot of work to do fleshing out our test suites for this one)

Secondly, it'll start highlighting your code to show you what code is executed, which branches are only partially covered, and which lines are skipped entirely. For example (ignore the sickly color scheme - I need to work on that):

Code Coverage example

This shows how several of the if branches are only tested in one direction, while the "Faces" block is skipped entirely. That also shows some of the trouble with testing XPages-run code: the Tycho environment can't reproduce the XPages environment fully, so some branches aren't testable in that way. I haven't looked into the possibility of gathering similar data from JUnit for XPages, so perhaps that's possible.

For now, though, this will have to do. And, like with these other "code improvement" techniques I've integrated lately, there's a lot of potential tedium - juggling when to write a test to cover some code that will obviously always work just to improve the highlighting vs. just focusing on the low-hanging fruit - but I expect that it will be a nice addition to my workflow over time.

Change Is In The Air

Aug 26, 2016 5:41 PM

Tags: xpages domino

During last week’s MWLUG, there was a clear sense that things are a little different this year. Dave Navarre dubbed the technical implications “platform agnosticism”, while I geared my presentation towards the feeling that change is in the air.

This is not totally new. Red Pill Now cast aside the XPages UI layer and most of the assumptions of Domino development to move to a new level; PSC's presentations have long developed a polyglot tone, and this ramped up this year; and people like Paul Withers have been growing with tools like Vaadin.

It's not too important to dive into the specifics of market forces or the shifting sands of technology, and the platform defensiveness that we tend to wear as a cozy blanket doesn't serve anyone properly. Our beloved Domino app-dev platform has grown pretty long in the tooth and it doesn’t seem like it’s in for a revitalization.

The situation is, fortunately, an opportunity.

One of the things I hoped to convey in my presentation is that, though the prospect of learning some of the ever-changing array of modern development tools is daunting, it is also exhilarating and profitable both professionally and personally. As insane as the tangled web of server frameworks, JS optimizers, language transpilers, automation tools, dependency managers, and so forth may seem, particularly compared to the simple days of Notes client development, there is a great deal of good news.

The popular tools are awash in documentation, with clear examples for doing basically whatever you will want to do. There's also a lot of conceptual overlap and familiarity: if your tool of choice loses favor, it won't be a start from square one to learn the next. And it's not required to learn every single thing that comes along. If you build yourself a Java web stack using, say, Spring that does the job, it's not also necessary to learn every single new client-side JS app framework that comes along.

And, in the mean time, there's a lot of great work left to do with Domino. There are XPages applications in use and development, and these will go a very long way. Domino remains a very capable platform, and the path through XPages can be a very natural lead-in to other technologies, especially if you focus on the aspects that carry on: Java, data separation, REST services, and so forth.

For my part, I, too, still have great work to do on Domino and XPages, but I'm also expanding beyond it with eyes open. As I mention frequently, I believe that Darwino is the best path for a number of reasons. When I have the opportunity, I plan to start getting into the meat-and-potatoes reasons why and examples of how to actually use the thing. Time permitting, I hope to have a series at some point for converting my blog over to a Darwino+JEE application, and I'll share my process of picking my tools and replicating with its current NSF form as I go. If all goes well, it should serve as an example of taking an existing XPages app and transforming it into something new.

This is an opportunity, and it's an exciting time.


Postscript: This is the optimistic take, granted, and some people’s situations are a bit more dire. Admins, I imagine, are in a strange spot (I hope you’ve been brushing up on ancillary tools!), and a lot of companies are doing a lot of hand-wringing about the future for app dev and maintenance as well, particularly those with a heavy Notes-client dependency. My point is that it’s not necessary to get too mired in the doom and gloom.

Change Bitterness and Accidents of History

Jun 17, 2016 3:16 PM

Tags: domino

It's pretty easy to see that change is in the air for Domino types. It's been taking a number of forms for a while now - the long delay since the release of 9.0.1 and associated aging of the tools and infrastructure have led to a series of forced adaptations for developers and administrators. Developers, for example, have had to keep light on their feet to adapt to new browsers and devices that the framework doesn't automatically support, as well as a shift toward manually including jQuery and other tools that have a bit more wind at their back than Dojo. Administrators, for their part, have had a series of heart attacks related to SSL and other security matters, usually involving a a lot of noise followed by (unfortunately, I feel) a good-enough patch from IBM.

That sort of thing isn't likely to get any smoother. In large part, that's entirely distinct from anything IBM does: the genie's out of the bottle when it comes to fast-moving platforms, and the best we can hope for is a sort of still-moving linga franca that can be sort-of-stable on the majority of targets. But then part of that is Domino: for all its virtues, it hasn't adapted for the modern world, no matter how much some of us would have liked it to. And that's had some negative side effects on us as a customer base, side effects that manifest as a gut-reaction rejection of the modern ways of doing things.

Take the SSL thing, one of my soap boxes: though the immediate problem could be summarized as "IBM should update their SSL stack", the larger issue it exposed was that our beloved monolith is mortally vulnerable to a single component falling behind. And, in fact, it made clear that we're spoiled by the approach: many of the reactions were basically that we shouldn't have to worry about things like reverse proxies or the general notion of distinct systems for web front-ends and the app server. And that initial rejection of the hassle has implications, limiting the average Domino installation's capacity to scale for load balancing or failover in a smooth way, things that come almost "for free" with a reverse-proxied setup common among almost all other app servers.

Developers have it worse: the "power user turned developer" history of Notes and the particular peculiarities of the platform* have left us desperately behind the baseline for modern development. The tooling has coddled us into preferring inline scripts and procedural programming to structured code organization, into viewing a thoroughly-staid language like Java as something outrageously complex, of only begrudgingly adopting SCM due to the platform-induced hassle, and of almost entirely ignoring automated testing. And similarly to the SSL discussion, the historic "one tool for every job" nature of Domino leads to natural pushback when faced with other platforms.

And I get why! And I feel it too. It's a real PITA to now always have to be on the lookout for some new point release of iOS or Android to break drop-down boxes or something, as opposed to years of deploying Notes apps that looked and worked identially across every version forever (pretty much). It's also a drag to run into situations where the problem is "easy" on Domino but more cumbersome elsewhere, like platforms that farm out their FT indexes to distinct servers, or don't include document/record-based security. That makes it very easy to become blinded to the tradeoffs, though. It may be nice that Domino is a one-stop-shop for so much, but it's a shop that requires Designer, that makes it very difficult to use third-party-dependency systems like Maven (even within Maven projects), that lags in DB features found elsewhere, that is only awkwardly accessible from other app-dev frameworks, that has an API that's a bit older than Windows Me, and that essentially never showed up for the modern development conversation.

Domino has always had a lot to recommend it, and XPages has carried us very far. And hey, this is enterprise software - even if there's never a major new version, there'll be paying work forever. It just may not be the kind of work you want to do, and it is almost definitely not truly healthy for the companies paying for it. So what I recommend is that you have a plan. The good news is that there are a great many next steps that build smoothly on existing Domino knowledge and, potentially, infrastructure. Certainly, I'm thoroughly biased in the direction of Darwino, but that's one of many. You could also do worse than learning a mature platform like Ruby on Rails (heck, you could run that on JRuby on Tomcat or WebSphere). Take some time to learn about reverse proxies and modern web-server setups. Basically: something. Just do it with an open mind, and don't balk at the first thing that's more complicated than the Domino equivalent. I think a bit of that will serve you very well.


* And, to be fair, of the overly-conservative nature of enterprise programming.

Other than, of course, being one of the progenitor NoSQL databases.

Platform Defensiveness

Jul 18, 2014 8:50 PM

Tags: domino

If you were a Mac user in the 90s and early 2000s, life could be tough. Though you loved your platform of choice and could see its advantages plain as day, you were in a distinct minority. The world was full of people ready to line up to explain why you're an idiot on a sinking ship. And the part that stung was that they weren't always wrong: for every person talking out of their ass about how great having a menu bar in every window was or the joys of running ancient line-of-business DOS apps, someone else pointed out the very real stability and performance issues that plagued classic Mac OS and OS X respectively, the fifth-class-citizen status among game developers, or the very-real possibility that Apple would join Netscape and Be under the treads of Microsoft's then-indomitable war machine.

The recourse for users was reflexive defensiveness. Mac market share is down to 2%? Well, that's the good 2%. Windows 2000 is actually stable and fast now? Well, the UI is still second-rate. Schools are switching en masse to PCs? Boy, they'll regret that when their cheap Dells give out! Much like the slings and arrows thrown at the Mac community, these defenses contained just enough truth to soothe the wounds, but it didn't matter. Maybe the community circling the wagons staunched the bleeding a bit, but the thing that really mattered was Apple breaking its own malaise and making great products again.

Unsurprisingly, what has me thinking about this is Domino. Specifically, the popularity of a post on Chris Miller's blog about Denny's apparently switching away from Notes. Now, I don't have a gripe with the post or the specifics of the situation - for all I know, Denny's is making the worst decision of its corporate-IT life. What bothered me is the paint-by-numbers way the story echoes through the Domino community. We've seen these things go around before, and the components are familiar when Company X decides to ditch Notes/Domino for Competitor Y:

  • "Just wait until they see the REAL cost of Y! Then they'll be sorry!"
  • "They must not be thinking about all their crucial apps! Y doesn't do that!"
  • "Their stated reasons are wrong! They said Domino doesn't do Feature Z, but Domino is actually the BEST at Feature Z!" (nowadays, replace with "Use some other IBM product to get Feature Z!")
  • "This is the work of some clueless IT manager who's just following the trends!"
  • "They must be using an ancient version of Notes! Modern Notes clients are exemplars of memory efficiency and clean design!" (alternative for server admins: "They must not have configured their mail environment properly!")
  • "I betcha they'll still be using Notes for their crucial apps in ten years, 'migration' or no!"

Much like the defenses of classic Mac OS, these all contain kernels of truth, and maybe Company X would indeed be better off sticking with Domino. But are these really claims you want to hang your hat on? The last one in particular is telling. Does it really fill your heart with pride that companies are going to be saddled with ancient, un-migratable code until the sun goes nova? There's a reason why "modernization" is such a big topic for Domino developers: the unfathomable mass of legacy Notes client apps is a severe issue. Whether you attempt to deal with it via clever en-masse approaches, by using remote-desktop workarounds, or by dragging apps one-by-one into the present, there's no getting around the fact that depending on Notes client apps is now an undesirable condition for a company.

So what's to be done about it? Well, for the most part, that's IBM's job. But for an individual developer, it's better to focus on why you should build a Domino app today, not decades ago. It builds on the past, sure - an XPages app I've been building for a client brings together data from dozens of their existing apps built over years in ways that would be much more difficult on other platforms. But even better is to acknowledge reality with clear eyes. Google Apps are really good, and they work on everything! Exchange provides a better mail/contacts/calendar experience for varied clients than Domino does. Non-XPages web-dev environments are brimming with surprising features and deployment has gotten good in recent years. The advantages of other platforms are not necessarily enough to be worth a switch, but they still exist.

Personally, yes, I care whether Domino ends up flourishing, but I profit more from addressing reality as it is rather than immediately dismissing bad news as illegitimate.

The Trouble With Developing on Domino

Jul 8, 2014 10:57 AM

Tags: domino

The core trouble in my eyes with developing on Domino is that it is unloved for what it is. Not so much by customers (I have no interest in whether companies use Exchange or SharePoint or whatever), but more by IBM. The situation reminds me of Apple with products like WebObjects, Xserve, and Aperture: there's clearly a dedicated and talented team behind it, but the organization's heart isn't in it. The analogy is inexact: first, Apple's overarching motivations and strategies are much easier to grok than IBM's; second, enterprise software never really dies - it just goes all COBOL and lingers in maintenance somewhere forever (did you know OS/2 is still a thing?).

So Domino is still around, and still exists, but is generally positioned as something for large companies to use for mail, or continue using if they happened to start doing so decades ago, or as an adjunct to Connections. But it's like with Lotusphere's rebranding: when (dev conference) + (biz conference) = (biz conference), the algebra to figure out the perceived value of (dev conference) isn't difficult. To a certain extent, this is just how IBM does business: they talk to large organizations' higher-ups and make their sales that way, not by being outwardly compelling. However, on this front, Bluemix's entry has provided a telling counter-example: though there's still the stock over-puffed business page, the bluemix.net site talks directly to developers using a layout and writing style that an actual human being might enjoy. They even have honest-to-goodness Developer Evangelists!

It leaves us, as Domino developers, in an awkward position. We have a full-fledged NoSQL server with an integrated Java dev stack, albeit one without a guiding soul. We have an easy-to-install, flawlessly-clustering, cross-platform server that is perpetually 20% away from being perfect for the modern world. Large portions can be charitably characterized as having been in maintenance mode for a long time: the core API, SSL in all protocols, calendar/contacts connectivity, indexing, the admin client, and so forth. The bad news is easy to perceive: less vendor interest and customers driven primarily by inertia make for a poor career path. But the point of this post isn't doom and gloom! The way I figure it, there are a number of bright sides to explain why I and others continue to develop on this platform:

  • There's always a chance that Domino will be less "Xserve" and more "Mac Pro", a product seemingly at death's door for years until it was given a fresh breath of life.
  • As Paul Graham pointed out years ago, when you're writing server-run software, you can use whichever platform you'd like. Though, all else being equal, it's nicer to have a popular platform, if you can use the tool to do greater work with your time than you would elsewhere, it's worthwhile.
  • Due to Domino's nature, almost all of its (technical) problems are solvable even if IBM never touches it again. Some of them, like the API, are things where the community has already built a workaround. Some, like SSL for HTTP, have seen IBM package their own workaround with the server. And the rest, like NIF and CalDAV/CardDAV support, linger as perfect "if I had time" projects.

The last one is crucial for our position: though a totally unsupported problem would eventually fall prey to incompatibility with newer operating systems and machines, Domino has reached a point in the last couple years (thanks to the extensibility API in 8.5.2 and the vital bug-fixes in 8.5.3 and 9) where it's an agglomeration of replaceable parts. There are enough hooks and APIs to - with varying degrees of difficulty - take advantage of the core strengths of the platform while dramatically improving the components on top. That's not enough to last forever, but it doesn't have to be. Apps built in the mean time will still run and improve, and any time spent programming on the platform is valuable experience that can be applied elsewhere, whether directly or more generally.

Just don't ask about licensing.

Another Round of URL Fiddling

Jun 19, 2014 5:59 PM

Tags: domino urls

I was working on another installment of "Be a Better Programmer", but the result isn't where I want it to be yet, so it'll have to wait. Instead, I'll share a bit about my latest futzing around when it comes to improving the URLs used in an XPages app. This has been a long-running saga, with the latest installment seeing me stumbling across a capability that had been right under my nose: the ViewHandler's ability to manipulate the URLs generated for a page. That method worked, but had the limitation that it didn't affect navigation between pages.

As it turns out, there's a slightly-better option that covers everything I've tried other than navigation rules specified in faces-config.xml (but who uses those in XPages anyway?): the RequestCustomizerFactory. This is one of the various classes buried in the Extensibility API and present in the XSP Starter Kit, but otherwise little-used. In the Starter Kit stub (and in a similar commented-out stub in the ExtLib), it can be used to add resources to a page without needing to specify them in code - say, for a plugin to add JS or CSS libraries without being specified in a theme.

However, there are other methods in the RequestParameters object that would appear to let you shim in all sorts of settings on a per-request basis - different Dojo implementations, different theme IDs, whatever "debugAgent" is, and so forth. The one that interests me at the moment is the UrlProcessor. This is aptly named: when a URL is generated by the XPage, it is passed through this processor before being sent to the browser. This appears to cover everything other than the aforementioned stock-JSF-style navigation rules: link value properties, form actions, normal navigation rules, view panel generated links, etc..

Following my previous desire, I want to make it so that all links inside my task-tracking app Milo use the "/m/*" substitution rule I set up for the server. The method for doing that via a UrlCustomizer is pretty similar to the ViewHandler:

package config;

import java.util.regex.Pattern;

import javax.faces.context.FacesContext;

import com.ibm.xsp.context.RequestCustomizerFactory;
import com.ibm.xsp.context.RequestParameters;

public class ConfigRequestCustomizerFactory extends RequestCustomizerFactory {

	@Override
	public void initializeParameters(final FacesContext facesContext, final RequestParameters parameters) {
		parameters.setUrlProcessor(ConfigUrlProcessor.instance);
	}

	public static class ConfigUrlProcessor implements RequestParameters.UrlProcessor {
		public static ConfigUrlProcessor instance = new ConfigUrlProcessor();

		public String processActionUrl(final String url) {
			String contextPath = FacesContext.getCurrentInstance().getExternalContext().getRequestContextPath();
			String pathPattern = "^" + Pattern.quote(contextPath);
			return url.replaceAll(pathPattern, "/m");
		}

		public String processGlobalUrl(final String url) {
			return url;
		}

		public String processResourceUrl(final String url) {
			String contextPath = FacesContext.getCurrentInstance().getExternalContext().getRequestContextPath();
			String pathPattern = "^" + Pattern.quote(contextPath);
			return url.replaceAll(pathPattern, "/m");
		}

	}
}

The method of adding this customizer to your app directly is, however, different: instead of being in the faces-config file, this is added as a "service", as mentioned in this handy list. For an in-NSF service, you accomplish this by adding a folder called "META-INF/services" to your app's classpath (you can add it in Code/Java) and a file in there named "com.ibm.xsp.RequestParameters":

In this file, you type the full name of your class, such as "config.ConfigRequestCustomizerFactory" from my example. You'll likely have to Clean your project after doing this to have it take effect (and possibly in between each change to the code).

Now, my current use is very limited; the UrlProcessor has free reign for its transformations and is passed every URL generated by the XPage (except hard-coded HTML, unsurprisingly) - you could change URLs like "/post.xsp?documentId=foo" to "/blog/posts/foo" without having to code that explicitly in your XPage. You could write a customizer that looks up all Substitution rules for the current Web Site from names.nsf and automatically transforms URLs to match. You could go further than that, trapping external links to process in some way in the "processGlobalUrl" method. The possibilities with the other methods of the customizer go much further (and are largely undocumented, to make them more fun), but for now just fiddling with the URLs is a nice boon.

Pretty URLs: A Step in the Right Direction

May 9, 2014 8:20 PM

Tags: domino

A long time ago, I mused a bit about the URL problem in XPages. The core problem then, as it is today, was that XPage URLs are ugly as sin. Domino URLs were never pretty, but XPages took them back a few steps. Just look at this normal-case monstrosity:

https://someserver.com/foldername/app.nsf/somepage.xsp?action=openDocument&documentId=E0023409DE744F8085257CD3007006A3

Barely any part of that has anything to do with the task at hand! Some parts - like the folder path and NSF name - are only tangentially related, while others - like "action=openDocument" are little more than an implementation detail. The inhuman UNID gets a half-pass by virtue of its value as a cluster-friendly identifier, but it would be better as a human-readable unique key.

What you'd REALLY want would be a WordPress-and-others-style URL like this:

https://someserver.com/blog/2014/5/9/pretty-urls

Now THAT'S a functional URL: every part of it is both explicable to humans and useful to computers as a unique identifier. More importantly, it establishes a clear hierarchy: any part of it can be traversed to get a conceptually-useful URL (assuming your app implements it). "/blog/2014/5" has a clear meaning: "give me all entries in the 'blog' app for May 2014".

Domino provides little assistance in getting to this point. There are web rules, yes, but that leaves two crucial problems: 1) your app needs to know about these rules, so it generates nice URLs and not eldrtich horrors and 2) you need to set up a server-level config for every web site + function combination. You can't just say "send all requests for /blog to so-and-so app" and then have that app handle everything past that. This is something of an impassable brick wall: Domino's original URL router is still in full effect even in the most modern of apps, and so the XPages side of an app doesn't even hear about a URL request unless it's in the form of "app.nsf/somepage.xsp" or "app.nsf/xsp/whatever" (and the latter form is something of a Wild West full of resource providers, servlets, and who-knows-what else).

But there's a twinkle of hope: because you specify all URLs in an XPage as relative to the app, that means that there's a post-processing service in there that translates a URL like "/foo.xsp" into a full server-relative URL like "/somedir/app.nsf/foo.xsp". That service has a name: ViewHandler. ViewHandlers have been one of my preferred tools in XPages ever since I discovered their utility in instantiating my soon-to-be-renamed "controller" classes. In addition to their role in creating pages and potentially intercepting page requests, ViewHandlers serve a crucial role: providing resource and action URLs. In XPages/JSF parlance, an "action" URL is what shows up in the form tag on the HTML result, while a "resource" URL is, well, basically everything else. When a control on an XPage specifies a URL, it passes it through the resource method to get the "real" URL. Normally, these are turned into the normal ".nsf" paths, but there's no reason they have to be. Here's an example of the two methods in my project-tracking app (named "Milo"):

@Override
public String getResourceURL(final FacesContext context, final String resource) {
	if(!isGlobalResource(context, resource)) {
		// Then switch it to "/m/whatever"
		return "/m" + resource;
	}
	return super.getResourceURL(context, resource);
}
@Override
public String getActionURL(final FacesContext context, final String pageName) {
	return "/m" + pageName + ".xsp";
}

I set up a substitution web rule on the server to translate "/m/*" to the full path of the DB. Because of this change to the view handler, now every <xp:link/>, <xp:image/>, theme resource, etc. uses the "pretty" path. Assuming I don't run into any major down sides, this is a huge piece in the puzzle! Now I can write my apps normally - referencing XPages, resources, etc. using the same portable, app-relative syntax as usual - and then let my ViewHandler determine if it should clean up the URL. This is a nice step above my second solution, which required either a special EL pattern or a custom control to translate the URLs; now that job can be passed up a layer of abstraction.

There are still two problems I know of standing in between me and using this everywhere: automatic detection of server support (e.g. reading names.nsf for matching web rules) and navigation between pages. Though the ViewHandler covers normal links and form action URLs, it doesn't handle navigation between pages based on navigation rules. I don't expect that to be an eternal problem, though; JSF has hooks for this sort of thing, so it should be a matter of figuring out how to use it there.

Overall, it's still something of an ugly solution: where other platforms have clean routing configs, XPages has a hodgepodge of server-level settings and in-app shims. However, I hope to turn it into a worthwhile combination of automatic configuration (by reading names.nsf) and clean declarative settings that I can build into the Scaffolding project.

Domino the Identity Server

Feb 11, 2014 9:10 AM

Tags: domino

As seems to happen a lot lately, my fancy was struck earlier by a Twitter conversation, this time about the use of Domino as a personal mail server. Not only do I think there's potential there, but it should go further and be a drop-in replacement for personal mail, calendar, and contacts storage.

I think there's tremendous value in controlling your own domain and the services on it, without being permanently attached to someone else's name for an email address (your school, your company, your ISP, Google). This is good not only for personal freedom, since it lets you pack up and move at will, but also for security, since a large third-party mail service is a particularly juicy target.

Unlike Domino's inherited-but-abandoned place as the preeminent NoSQL server and replicating app-dev platform, Domino is just barely shy of being this server. You could already hit the three main services reasonably well by using the Notes client or iNotes, but actual humans shouldn't have to do that. It's already (more or less) there for mail with IMAP, while support for CardDAV for contacts and CalDAV+public iCalendar feeds would round it out for the other two pillars. Technically, the only things standing in between the existing open-source WebDAV plugin for Domino and this imagined future are the complexities of plugin development and the RFC.

The other main aspects that could make this great are further refinements to Domino's existing capabilities: a bundled spam filter (say, one of the open-source tools that can do the job already) and a strong configuration focus on creating an SSL-secured public-facing server. Non-SSL variants of IMAP, POP (if you must - that could be removed entirely), and HTTP should be off by default and the configuration should encourage you to acquire SSL certificates with your own private keys (the Server Certificate Admin would need a revamp for proper key size and ciphers), as well as S/MIME certificates to tie with each user for signed/encrypted mail outside the Notes client. Though Domino's history with the NSA is... checkered, it's still remarkably well-positioned to provide a secure foundation to deter snooping eyes. Certainly, running a Domino server own your own or a rented/virtual server is leagues better than a fully-managed service in this respect.

Having those features in place and smoothly integrated with a nice setup assistant would make for a very compelling product: an all-in-one, easy-to-install server that runs on several modern OSes and handles secure replication across physical locations at already-actually-affordable prices. Admittedly, I don't know how compelling that product would be for IBM's accountants, but it's certainly compelling for me, and the world could use more decentralization like this.

How I Want To Use Domino, Take 2

Oct 30, 2013 4:10 PM

  1. How I Want To Use Domino
  2. How I Want To Use Domino, Take 2

A while back, I wrote a post about how I wanted to use Domino. The gist of that was that I was enamored with the idea of using Domino as a back-end database, but not necessarily as a app-dev platform on its own - basically, how you would use a SQL or NoSQL database. Since then, I've doubled down on my use of XPages as an app-dev platform with many advantages, but I still find it very useful to imagine Domino not as "Notes apps on the web, now with a modern coat of paint", but as a collection of related but not mutually-required components competing with other web-dev stacks.

I started working on an updated take on that post, inspired by some recent posts and chat conversations I've had (and I may return to it), but then I ran across Grand Decentral Station, which is a vision of an OS/app-dev platform taking the best of the lessons of the last decade and turning them into a coherent platform. It's a compelling vision, and reading it made me realize something: most of those goals are describing Domino, or, more accurately, the Domino that could be. Take a look down the list and see how many points Domino goes about 80% towards:

  • App Installer & Updater: though Domino doesn't really handle app versions, the deployment strategy is nonetheless quite good, with all app code contained in a distinct NSF, not a bunch of files strewn in a couple directories.
  • Sandboxed apps: again, Domino doesn't quite sandbox apps, but appropriate use of ACLs can bring you close.
  • Mail server: I hear tell that Domino is capable of acting as a mail server.
  • Calendar server: with proper CalDAV support, Domino could act as a real calendar server for non-Notes clients like OS X and iOS.
  • Addressbook server: like with the calendar server, this is just a matter of adding CardDAV support.
  • Asset handling: CSS and JS optimization in XPages is a huge step in this direction.
  • Avatar server: the Directory can already act as a profile-picture server for Sametime, and something like mypic could standardize this use.
  • vCard server: well, it already serves LDAP.
  • Unified sessions: done.

And to cap this off, the prescribed per-app database is CouchDB, which is already modeled on Domino. And, of course, it already has a standard API for email, which conveniently doubles as a method of cross-app messaging in some cases, and its replication and clustering are top-notch. It's not, itself, an OS, but its fairly-cross-platform nature means that that problem is already "solved": install Domino on the server platform you're most comfortable with and it acts basically the same.

Of course, that final "20%" is, as always, the crux of the problem. Domino is only really a fully-integrated mail/contacts/calendar server when you use Notes or iNotes, XPages and legacy Domino dev are really the only games in town if you want to maintain the benefits of the NSF package, agents aren't integrated with the XSP environment and DOTS isn't a real replacement yet, there are still a number of items on the list not at all touched on by the existing stack, and licensing basically removes Domino from consideration for app development for anyone not already mentally invested in it. But hey, one can dream, no?

Domino Wishlist, September 2013

Sep 20, 2013 1:06 PM

I joined in a tweet from Paul Hannan earlier about desired Domino features (that's a very worthwhile thread to read - lots of great ideas from others) and it's had me thinking about more on my list. Fortunately, I have a blog for this kind of thing! I'm leaving off a couple big-ticket items like Eclipse-4.x-based Designer on the Mac because IBM may not be required for that.

  1. Make $WSIS not trigger crummy SSL behavior (really, first-class support for proxies in front of Domino, not just IHS/Windows)
  2. Direct access to note item data via Java
  3. Newer JRE
  4. Direct creation/update of NSFs from git server-side
  5. New XSP model-object framework
  6. New views, though that may be our job
  7. Aggressive, small-company-friendly licensing to compete with OSS stacks
  8. NSF-based SSL keychains
  9. NSF-based jvm/lib/ext distribution
  10. "Sensible defaults" mode for clean-install servers to get appropriate memory/cache/performance settings on modern setups without learning dozens of secret notes.ini settings
  11. apt-get-based distribution and updates for Domino on Debian/Ubuntu
  12. Improved XPages file-upload and -download controls to handle modern file uploading and provide better hooks for non-dominoDocument bindings
  13. Bootstrap theme for XPages. Done - thanks!
  14. Built-in default support for authentication fields in LDAP
  15. init.d startup script installed with Domino, because seriously, who ships server software without one?
  16. OS X build of Domino 64-bit
  17. Built-in Firefox sync server, because that would be a nice overture to general open-source use
  18. Support for HTTP authentication filters in Java
  19. Built-in beer database to compete with Couchbase's substantial lead in the area
  20. Easier configuration of multiple TCP/IP adapters on a server and assignment of services to bind to them
  21. Full control over URL routing inside a database (e.g. "no legacy" mode where a Java class can map requests to resources)
  22. Keep up the ODS improvements - 8.x was a great set of releases on this front
  23. CalDAV and CardDAV support - Domino should be a close-to-drop-in replacement for Gmail on OS X and iOS
  24. Along the lines of #7, a general push to establish Domino as a top-flight NoSQL database and app-dev platform, not just "your old company's mail system"
  25. A real focus on standard-API performance, with less XSP-specific cheating

I'm sure this list is not exhaustive.

The Greatest Domino Poster of All Time

Apr 20, 2012 7:23 PM

My company is moving out of the offices it has inhabited for... longer than I've been alive, basically. During this move, we've had plenty of opportunities to come across relics of its past form as an instructor-led training location: old courseware, ancient versions of Windows, NetWare, and Notes/Domino (from before it was Domino), and some priceless marketing materials. Some of the posters we've found are worth a good chuckle or two (a NotesMail poster boasting "Finally, a mail system you will never have to exchange"), but this one is my absolute favorite thing related to Domino:

Amazing Domino poster

It's amazing, like a Domino ad out of Yellow Submarine. What I wouldn't give to have IBM's marketing to be more "Then, suddenly, bursting from a magic lotus blossom" and less "The Social server software for Socially Socializing with Social people in a Social way - just ask this guy in a tie."

Fun fact: the hooded "executioner" guy in the top right being punched out by Domino is labeled "SuiteSpot". SuiteSpot, it appears, was some sort of Groupware/web/mail/directory offering from Netscape. I guess Domino won that battle.

File Under "Man, I Hope Designer Still Works After This"

Apr 8, 2012 6:15 PM

Tags: domino

Well, I think the new blog has gone relatively smoothly, other than my accidental re-posting of my SQL-migration post (which oddly seems more popular than its first run, just one day earlier). That means it's time to get started on the next phases.

Other than the mundane setting-up-a-blog stuff like implementing search and proper draft posts, I have a lot of work to do surrounding my Ruby bindings. Since I'm going to eventually want Ruby "script libraries" and other handy non-inline uses, I'm going to need a proper Ruby editor. TextMate-via-WebDAV is OKAY, but not exactly elegant, and it involves too much overhead and awkwardness to be a great solution. Better would be to get DLTK - which has Ruby syntax highlighting - working in Designer.

On my first attempts, I ran into problems where using DLTK's update site demands an updated version of Eclipse, which could mean disaster for Designer. However, it turns out that GLTK has been part of mainline Eclipse for a while, so I figured I'd check to see if I could get to it via another route. Evidently, Designer 8.5.3 is based on Eclipse 3.4 (Ganymede), so I added Ganymede's update site to Designer's plugin installation list and, lo and behold, Ruby was there in the "Programming Languages" section. It still required the 3.4 version of org.eclipse.core, but, having backed up my Windows VM for this purpose, I felt daring. I clicked "Select Required" to select all applicable dependencies and hit Yes as many times as it took to finish the installation. After rebooting Designer, I found that Ruby scripts are now all fancily colored:

Ruby in Designer

If you'd like to try this yourself, I strongly suggest you do as I did and make a backup copy of your Windows VM first (or do whatever it is people who use Windows as their host OS do when they're about to ruin their system). Designer can be very picky about what plugins you install.

Making the Dogfooding Switch

Apr 7, 2012 11:55 PM

Tags: domino blog

I've finally done it: I've switched my blog over to Domino. I did it for a couple reasons:

  • To silence the voice in the back of my head constantly saying "why are you using WordPress? You're a freaking web programmer! Write your own!"
  • To put my Ruby-in-XPages code through its paces in the way only a live site can.

I've already had to fix a couple holes in my Ruby adapter, mostly revolving around the fact that I haven't bothered to properly handle serialization and JSF's StateHolder interfact. For now, I've patched the worst problems, but this gives me a great reason to come up with a proper solution. To test it out, I'm making sure to avoid "#{javascript: ...}" code blocks in favor of using plain-EL and "#{ruby: ... }" exclusively. So far, the only really awkward parts are the giant swaths of yellow-squiggly "I don't understand this" underlining in Designer and having to put xp:repeat values on xp:dataContexts rather than writing the computation inline. Not too shabby.

The whole thing's a bit shoot-from-the-hip at the moment, since it's only existed for a day. There's no search (though that'll be easy), the site design is from my old college-era blog, and I have to write the posts' HTML by hand in the Notes client (you know, like on all modern blogging platforms). But hey, I have the old posts in there, plus an archives list and, as long as it doesn't mysteriously die again like it did a minute ago, Akismet-backed commenting.

So let's see how this thing works! Don't be surprised to see the WordPress version again if things go catastrophically wrong.

Import from SQL to NSF: It's So Easy!

Apr 6, 2012 7:57 PM

Tags: domino sql

I decided I should probably finally get around to moving this blog from WordPress to Domino, if for no other reason than to have a perfect testbed for the weird stuff I've been doing lately. The first task is to write an importer, so I decided to just do a straight SQL rows -> Domino documents import. This couldn't be easier if you follow this simple guide:

  1. Write a Java agent that uses the appropriate JDBC connector for your database. In my case, it's MySQL, so I had it do a "show tables" to get a list of tables, then loop over those to do "select * from whatever" statements to get all documents.
  2. Since it's Domino, you can just do doc.replaceItemValue(column, rs.getObject(column))!
  3. Oh wait, you can't all the time. Make sure to handle a couple cases, like converting BigIntegers to longs and Timestamps to DateTimes.
  4. Wait, Domino doesn't even handle Boolean? For frack's sake, FINE: row.replaceItemValue(column, cell.equals(Boolean.TRUE) ? 1 : 0);
  5. Oh hey, that import went really smoothly! Now I'll just make some views based on the table name and... crap. Forgot to include the table name! Maybe I shouldn't have written this importer at four in the morning.
  6. Better delete all the documents and start over.
  7. What's this, an error when I went to delete all the documents in the All view? "Field is too large (32k) or..." oh no.
  8. Oh crap.
  9. Crap!
  10. Ah, I know - I'll just write an agent to do session.CurrentDatabase.AllDocuments.RemoveAll(True) and that'll fix that.
  11. Hmm, nope, that didn't work. Alright, based on the documents created around the same time, I can guess that it's the "option_value" field that's too big. Why did it even cram that data into summary if it completely breaks the document? Well, no point in dealing with that now. It's time for StampAll to take care of that!
  12. Nope? Okay, how about session.CurrentDatabase.GetView("All").AllEntries.StampAll?
  13. Not all categorized? What kind of error message is THAT?
  14. Time to delete the database and start over!
  15. Alright, THIS TIME, check to see if the String value is over, let's say, 16000 bytes and, if so, store it in a RichTextItem instead.
  16. Oh nice, it worked.
  17. Oh crap, I forgot to include the table name again.

And that's all there is to it!

#{ruby: 'it\'s a start'}

Apr 3, 2012 7:55 PM

Tags: domino ruby

Oh man, I think this might actually work. Feeling adventurous this evening, I decided to look into the XPage runtime's expression language handler. After poring through tons of methods, interfaces, implementation classes, EXTENDED implementation classes, and disparate JARs, I narrowed the prefix handler down to the "FactoryLookup" property of the IBM-specific variant of facesContext's Application. With that, which is basically a hash, you map a prefix to a handler factory (it's always factories with Java, isn't it?). Once there's a handler registered, you can then bind expressions with that prefix.

This actually means what it implies:

<xp:text value="#{ruby: 'hi from ruby'}" />

And it works! Better still, the show-stopper in Designer seems to have been resolved in one of the recent versions: rather than error'ing out when it sees an unrecognized EL prefix, it nags you with a warning but otherwise proceeds without problem, saving the script as-is in the resultant Java.

Now, this is VERY much a first pass at the idea - all I did was write extremely skeletal implementations of the Factory and Binding classes and then I register them in the beforePageLoad event of a page:

var app = facesContext.getApplication()
var facts = app.getFactoryLookup()

var rfac = new mtc.ruby.RubyBindingFactory()
facts.setFactory(rfac.getPrefix(), rfac)

I can't even begin to stress how not the right way this is. It doesn't work in all controls (repeats, for example, seem to do an extra syntax check), it doesn't yet have any context from the surrounding environment, and, most importantly, who knows what horrible things it's doing to the HTTP stack?

Still, the crucial point is that it really, really looks like it can work.

The two classes I wrote, which no one should, under any circumstances, use, are here: https://github.com/jesse-gallagher/Domino-One-Offs/tree/master/mcl/ruby

Putting the Domain Catalog to a Bit of Use

Apr 3, 2012 1:33 PM

Tags: domino

Since I kind of backed my way into Domino development and administration, there are a number of areas of the server's functionality that I'm either unfamiliar with or casually brushed off as unreliable or not overly useful.

The Domain Catalog is one such area: I've been vaguely familiar with it, but have never bothered to tend to it or use it to solve problems. Fortunately, a problem it's perfectly suited to fell into my lap. In an overarching administration database, I want to get from a document with a database doclink to the database itself as quickly as possible. The easiest way, programmatically, is to use the .ServerHint and .DBReplicaID properties in a call to an unconnected NotesDatabase object's .openWithReplicaID(...) method.

This works well, but doesn't take into account the existence of local replicas - the admin database is running on one server, while the databases are created and linked to on the primary production server. Given that these servers aren't even in the same physical state, it's significantly faster to use a local replica when available, and only fall back when it hasn't been created yet. My first attempt at getting around this was direct: I first call .openByReplicaID(...) with a blank first parameter and, if it's not open, try the real server. Again, this worked, but is not ideal. It's programmatically ugly and, besides, there are more than just the two servers. .openWithFailover(...) looked promising, but seems to only accept real file paths, not replica IDs, which ruled it out for this purpose.

This is where the Domain Catalog comes in. It already contains a (hopefully-up-to-date) list of all of the replicas of every database in the domain along with everything I could need to connect to them. Furthermore, I realized I could use this information with a dash of extra intelligence: since I know which servers are physically closest, I could use that to find the "best" replica in each situation. So I made a view for this admin server ("Ganymede") of all of the database stubs, sorted first by replica ID and then by a column with this formula:

name := @Name([CN]; Server);
@If(
     name="Ganymede"; 1;
     name="Demeter"; 2;
     name="Invidia"; 3;
     name="Dionysus"; 4;
     1000
)

With replica ID in hand, this means that I can just do a @DbLookup() in the view and the first result will always be the best, regardless of which server it is. I have another column with the full "Server!!FilePath" path, so I can just pull that value and do it in one function. Nice, easy, and it lets Domino take care of the nitty-gritty details for me.

What Makes the Hassle Worthwhile

Apr 2, 2012 10:34 PM

Tags: domino ruby

I've been toying with my Ruby servlet a bit this evening and it didn't take long to start having some fun. For example, here's a snippet from a page I'm building with Markaby, which is an aging little library that makes building HTML pages declaratively a cinch:

$database.views.sort { |a, b| a.name <=> b.name }.each do |view|
li { a view.name } unless view.name =~ /^\(.*\)$/
end

That prints out the names of all the non-hidden views in the database, sorted alphabetically, inside an HTML list. That's barely scratching the surface and not often useful, of course, but it's a proof of concept.. Server JavaScript can do some cool things, but the required syntax makes it a classless language by comparison.

I also realized earlier today that the same idea here can be translated to writing agents in Ruby by creating them as Java agents and putting the Ruby code in attached Resource files. It's a BIT of a hassle, in that I can't use the WebDAV trick to edit the scripts with TextMate, and it seems like I have to manually rebuild the agent whenever I change the script file, but that's not too bad. I'll probably try it out on some non-critical agents on my dev server to see if there are any critical memory issues.

Next step: figuring out this newfangled OSGi doohickey.

My Recurring Ruby/Domino Dream

Apr 1, 2012 6:54 PM

Tags: domino ruby

As is no doubt clear by now, one of my obsessions when it comes to Domino is trying to make my programming life better, and one of the best ways I can think of accomplishing that is if I can make it so I can program in Ruby instead of one of the godforsaken languages natively supported.

In general, my attempts towards this goal have fallen into three categories:

  1. Accessing Domino from Ruby as one might a normal database. The very-much-in-progress fruit of this is my Domino API for Ruby project.
  2. Using Ruby as an alternative scripting language for XPages, like "#{ruby: blah blah blah}". This would be nice, but I haven't made any progress towards it.
  3. Running Ruby scripts out of an NSF via the web, as one might a non-scheduled agent

The last one has been the one at which I've taken the most swings, and I actually made some progress along those lines today. I decided to give a shot to using Domino's ancient servlet support combined with JRuby to accomplish this, and I think I've done it.

Since this is Domino, there were a couple hurdles:

  1. Servlets aren't automatically authenticated as the current Domino user. When I was looking into this, I read a lot about using NotesFactory.createSession(null, token), where "token" is the value of the LtpaToken cookie in an SSO configuration, which I use. However, this didn't work for me, possibly due to using site documents. Fortunately, there appears to be a better way: NotesFactory.createSession(null, req), where "req" is the HttpServletRequest object passed in to the servlet's method. This works splendidly, giving me a session using the current user without having to worry about figuring out their password or dealing with tokens manually.
  2. Finding the context database and script. The servlet is triggered properly by ".rb" extensions in file resources, but that still left the problem of actually opening that resource. Fortunately, the Session class has a resolve(String) method that can be used to get a Domino product object from a Notes URL. For file resources, this uselessly gives back a Form object, but you can get its UNID from the getURL() method.
  3. Reading the contents of the script. File resources appear to store their data in a $FileData rich text field, but that doesn't make it easy to deal with. For now, at least, I've gone the DXL route: I use the Document's generateXML() method, find the BASE64-encoded value in the <filedata> element, and decode that with sun.misc.BASE64Decoder.
  4. Actually executing the script. I banged my head against this for a long time: I kept getting NoClassDefFound exceptions for org.jruby.util.JRubyClassLoader when trying to run the script. I assumed at first that this was a problem with the class loader not finding this secondary class, so I tried tons of stuff with moving the JAR around, repackaging my servlet, and even trying manual class loading.  However, the exception was a red herring: it was really a permissions thing, so, not wanting to take any guff from a JVM, I granted myself all permissions and it worked great.
  5. Editing the Ruby script. This part wasn't really a crucial flaw, but the fact that Designer doesn't know about Ruby meant that editing the file resource involved no syntax highlighting or other fancy features. DLTK doesn't seem to play nicely with the Eclipse version in Designer and I've ruined my installation by messing with that stuff too much in the past, so I decided to try WebDAV. Unfortunately, Domino runs the servlet on WebDAV requests too, so, rather than trying to figure out how to handle that nicely, I just switched to accessing the database via a clustered server that doesn't run the servlet.

It was a lot of hassles, but it seems to work! The cleaned-up version of the method I used is:

  1. Put the "complete" JRuby JAR into Domino's jvm/lib/ext folder.
  2. Put the compiled class file for my servlet into Domino's data/domino/servlets folder.
  3. Grant all permissions to the JVM.
  4. Enable servlets for the server in question and specify "rb" as one of the file extensions.
  5. Enable the servlet in servlets.properties, as in my example.
  6. Restart HTTP/the server.
  7. Add .rb Ruby scripts as file resources to a database. I made one that makes a list of some documents from a view in my DB.
  8. Open the resource in a web browser and see it work! Hopefully!

One caveat: though I set the error and output streams of the Ruby environment to be the HTTP output, this seems to not always work, so I've found it best so far to use "$response.writer.println" instead of "puts" for writing text to the stream. I'll see if I can fix that. Turns out writing to the HTTP output should be done by calling setWriter(Writer) rather than setOutput(Writer).

Additionally, that "$response" there isn't a built-in feature of JRuby - it's a variable I put into the runtime. I do this along with "$request", "$session", and "$database". Think of them like the equivalent variables in XPages.

Some questions you may have are "is it stable?" and "is it fast?" and to those I have a simple answer: beats me! I just got it to work a bit ago and I don't know if I'll ever even use it. It's quite exciting, though, and that's what really matters.

https://github.com/jesse-gallagher/Domino-One-Offs

In Between My Project and XPages

Mar 15, 2012 4:11 PM

Tags: domino xpages

Despite my grousing about the state of programming for Domino in general and Designer in particular, I'm still mostly a fan of XPages. I use it for my guild's web site and pretty much every new project at work. However, I haven't been able to crack migrating my main work database template over.

Without getting too much into it, the point of the template is to create one database per project to act as a project web site listing online events with arbitrary registration forms and exit evaluations (among other things). Except for custom changes, everything is done via the normal Notes client, not Designer - pages exist as documents with a Body rich text field and associated data and register forms/exit evals, crucially, contain a set of response documents describing their fields. It's important to keep everything as visual as possible, because the users (the people at my company that set up these web sites) are not programmers.

I scrupulously try to avoid modifying the design of the database, so I go out of my way to do custom work via the existing mechanisms as much as possible, and I make pretty good use of rich-text-isms like embedded views, tabbed/computed tables, attachments, buttons, and computed text. In the case of the generated forms, fields can be placed either automatically (with a surrounding template of HTML per-field) or directly into the rich text body via <%FieldName%> placeholders (creating an experience like form design in Designer). Additionally, I have "WebQueryOpen" and "WebQuerySave" fields in the documents for formulas that I pass on into @Eval() in the equivalent places in the actual forms; most of the time, I use these for running agents.

Computed Text

Of the various potential show-stoppers, I think computed text fares the best. For one, it mostly worked - the computed formulas are indeed evaluated and the result is put into place correctly. However, they don't have the same environment you get with the "classic" design elements. The big one that I ran into immediately was @UrlQueryString(...) - it appears that the rich text renderer doesn't inform the rich text about its web environment completely. Prior to 8.5.3, "Display XPage instead" pages didn't know about the real URL, so they couldn't get query parameters at all, but 8.5.3 appears to pass that information along properly. So that means it MIGHT be fixable, if I find a way to properly set fields in the document before the rich text is rendered, so I can set QUERY_STRING - if I can do that, either @UrlQueryString(...) will work or I can manually parse the string as needed.

Embedded Views

They work! ...ish. It looks like icon columns get their URLs a bit messed up, but I can't think of a time when I actually used them, particularly in a situation where I couldn't just write out the URL myself.

Tabbed/Computed Tables

From a cursory glance, I think I'd be SOL when it comes to these. Tabbed tables render flattened out and computed tables don't seem to work. The former can be improved on easily with Dojo, but the latter would mean replacing server-side business logic with client-side JavaScript, which would lead to headaches.

Attachments

These show up as "(See attached file: foo.jpg)". So... not functional. I might be able to fix it by using a filter on the text to replace the HTML with a link to the document, but I don't know if I could get the attachment image properly. Attachment images aren't amazing, but sometimes they do the job.

Buttons

Nope.

WebQueryOpen/WebQuerySave

In some cases, I could run the formulas through session.evaluate(), though I'm not even sure queryOpenDocument/postOpenDocument let me hook into the right spots (sometimes I set fields on document open). I don't think that would run agents, though, so I'd be stuck trying to parse out the text to look for @Command([ToolsRunMacro]; "...").

User-generated Forms

This is the toughest one. I've given this thought a number of times, and I can't think of a great solution. I can't just re-use the subforms I have already, I don't think I can generate and import an XPage via DXL (though I haven't given that a significant shot), and I can't just do a <xp:repeat/> to create all the fields, since some are in the body area. The main routes I can think of to try are a) arranging the fields on the page after the fact via JavaScript and b) generating the page components on the fly with Java or Server JavaScript. I'll probably give the latter a shot eventually, but I expect it to be a bag of hurt.

 

It's a lot of hurdles! I'd love to switch over to XPages, particularly since, for everything that's more difficult than using classic elements, there are 10 things that are way easier. It'd just be quite an investment of time to merely get up to par with the functionality I already have, if it's even possible in all cases.

Quality of Life: Eclipse Color Themes

Mar 10, 2012 6:21 PM

Tags: domino

Well, this is a nice quality-of-life improvement: the plugin Eclipse Color Themes, which (so far) works just fine in Designer (8.5.3). Having to go through every single editor type and manually pick each color for each code element every time I wanted to change a color theme was always a huge annoyance, particularly compared to other editors. Fortunately, that plugin handles it pretty well, though it unsurprisingly doesn't support LotusScript, so that's still manual. I installed it without problem, found its settings in General\Appearance\Color Themes, installed a port of my preferred theme, and saved myself tons of hassle.

How I Want To Use Domino

Mar 7, 2012 12:43 PM

Tags: domino
  1. How I Want To Use Domino
  2. How I Want To Use Domino, Take 2

From my perspective, there are three main problems with Domino: the limits, the client, and the server. Now, that's a lot of stuff... most of the product, in fact. However, the facts that I'm still programming for it and that my company's 16GB project-tracking database is as snappy as it was when it was empty attest to the core quality of the product. Off the top of my head, I can think of a number of things that make Domino salvageable:

  • Reader fields. These are hard to beat and hard to find elsewhere. While you can do everything without them that you can do with them, you'd likely spend a lot of time worrying about edge cases or limiting yourself to coarse security. If a user doesn't have Reader access to a document, it doesn't exist (well, except for some shadows in view indexes).
  • Solid, straightforward directory integration. Going along with reader fields, it's very helpful that the directory system is part of the overall product and works well enough with LDAP that you don't have to worry too much about it. User names, group memberships, and roles all make sense.
  • Full-text searching. It works and it's quick.
  • View indexing, more or less. As long as you don't rock the boat too much and don't think about what you used to do with SQL tables and views, Domino does a fine job of maintaining views for you.
  • File attachments. They work well and can even be full-text searched if they're the right type.
  • ID files. I used to hate these, and they're often more hassle than they're worth (as my C API programming is demonstrating), but they have some distinct advantages. If you squint, you can see them as really long password that may themselves have another level of password. They're also essentially the same concept as SSH keys. They also tie into handy encryption and certificate abilities that I don't usually have much use for.
  • Potential capability as a blob store. I saw a great post a while ago talking about the ability to just cram arbitrary data into a Domino document, and I think this is a very valuable line of thinking. If your data fits into the model of "a couple queryable bits and then a block of arbitrary data" or the "upside-down" method of memory-storage-first from the post, such a setup could work extraordinarily well.
  • Replication and clustering. Sync is not hard... unless you let your deletion stubs expire.
  • Agents, more or less. The language choices and lack of immediacy make agents kind of a pain sometimes, but it's still handy to have code directly associated with the database with schedule and event triggers.
  • Read marks. It's nice to have the option to let the server handle this for you when it fits your needs.

That's on top of the various benefits you get from document databases generally, like multi-value fields and the lack of schemas. Note, though, that none of these are (directly) related to Domino's chops as a mail platform, web server, or GUI app environment, its choice in programming languages, the value of its standard array of templates (though they can be handy), or pretty much any virtue extolled by its marketing materials. The guy writing on the invisible whiteboard on IBM's page doesn't care about reader fields or blob storage.

Though most of my Domino work is for the web and XPages are a better way to do that than the legacy elements, it's still a drag. Java as a language is a hassle (the platform has its charms), Server JavaScript isn't a full replacement in the way that Groovy and JRuby can be, and working on top of a giant Jenga tower of Java classes and abstractions can be hazardous to your health.

I really just want to treat Domino like MongoDB, CouchDB/Couchbase, and the rest: an environment-independent document database. This is probably not worth the effort, particularly since IBM seems passively hostile to the notion, but I find it to be a compelling idea. It doesn't have to fit into, say, Rails, but grinding Domino down to its solid NoSQL core could open up a lot of possibilities.

Some Niceties of Implementing a Notes API

Feb 20, 2012 7:56 AM

Tags: domino ruby

There are a couple things about writing my Ruby wrapper for the C API that make it particularly fun, mostly related to getting to add abilities that I desperately wish were there in the normal APIs.

  1. Ruby-style (forall) looping. Anyone who has iterated over a NotesDocumentCollection knows the drill: set a variable to the first element, start a while loop, and make sure to set the variable to the next one at the end. Writing it one time isn't so bad. Writing it hundreds of times, though? It gets to be a drag. Getting to write docs.each { |doc| ... } is a breath of fresh air.
  2. Easier design-elements-as-notes access. The normal API lets you get a NotesDocument version of a NotesView via its UniversalID property, but for everything else you need a NotesNoteCollection, which is a hassle. Since all design elements are Documents anyway, I've just made their wrapper objects subclasses of Document (though I may change that to just a #document method if it gets hairy) and I've put a #get_design_note method on Database that lets you find a design note by name and flag class (NIFFindDesignNoteExt).
  3. HTML and DXL everywhere. I use DXL fairly constantly (mostly for design elements), and for the most part it's a hassle. Not only do you have to create a NotesDXLExporter, but you also have to get the Document version of the design note you're dealing with and run through its process. Not impossible, but sort of a hassle. Now, though, I just put a #to_dxl method on everything - this is like the .generateXML() method in Java, but consistently applied. Similarly, I can't count the number of times when it would have been handy to get an HTML representation of some element, even if it was just the dated stuff that the legacy renderer puts out. Since that's all there in the C API, I just put a #to_html method on everything and a #get_item_html method on Document for very easy access to web-friendly versions of MIME and Rich Text items.

It's just a shame that I probably won't be able to use this: what I'd really want to do would be to use it like a database driver on a Ruby-driven web site, but the fact that it's so tied to ID files makes that tough. Still, it's great exercise to write it, and maybe I'll cave and set up some sort of multi-process hydra beast to suit my needs.

Started Work on a Ruby Wrapper for the C API

Feb 15, 2012 7:28 PM

Tags: ruby domino

As I had mentioned before, I've been tinkering about with the Domino C API, specifically with Ruby. Although I'm not sure I'll actually have a use for it (from what I can tell, the C API is very tied to ID files and threads, which would make a multi-user web server thing cumbersome), I've decided to go for it and write a wrapper for the API generally based on the Java/LotusScript API. This is serving a number of purposes:

  1. It gets me off my (metaphorical) duff and in front of a text editor during my off hours.
  2. It lets me work with Ruby in a real capacity. And moreover, it'll expose me to how to write a Ruby library.
  3. It's introducing me to proper version control, something I absolutely need to do better.
  4. It will eventually force me to write structured documentation, which I haven't had to do in the past.
  5. It should fill a hole currently only partially filled with OLE-based libraries, though the ID-file thing will make it awkward.
  6. It brings me back to the world of C structures and pointers, albeit cushioned by FFI.
  7. By the time I'm done, I'll know Domino inside and out.

So far, I have some of the basics working: creating a "Session" by passing in the program directory and the path to an INI file, fetching databases, traversing and full-text-searching views, reading view entries, and reading basic and simple MIME data from documents. Obviously, there's plenty more work to do, and I don't know if I'll bother implementing the more esoteric classes like NotesRegistration, but it's off to a fun start so far:

https://github.com/jesse-gallagher/Domino-API-for-Ruby

Point 7 on that list has proven surprisingly interesting. It's fun to see the bits that apparently never fully made it onto the product (like number ranges), the little optimized paths where the documentation goes out of its way to point out that it's particularly efficient, and all the data structure sizes that create the various limitations that make Domino programming such an... experience.

This Dynamic View Customizer Is Getting Into Shape

Feb 13, 2012 10:08 AM

Since last week, I've made two nice improvements to my dynamic view customizer:

  1. I added some support for twistie images when the referenced DB is on the same server. The code assumes that the referenced images are image wells with at least two entries, but I can't imagine why that wouldn't be the case in practice.
  2. I vastly improved my handling of color columns. Previously, I had been resorting to hacky methods like hidden <div>s read by JavaScript or surrounding <div>s styled to take up the whole cell, but those were terrible and easy to break. Now, though, I'm doing it right: the code adds a value binding for the column's style attribute to create the CSS for each cell, which is ideal.
  3. Empty categories now are translated to "(Not Categorized)", as in the client.

Now that my code is presentable (albeit oddly structured and uncommented), I figured I may as well toss it up on GitHub:

https://github.com/jesse-gallagher/Domino-One-Offs/blob/master/mcl/reports/DynamicViewCustomizer.java

To note if you want to use this in your own project: it references the "mcl.JSFUtil" object, which started as the mindoo object of the same name and has since turned into my bin for common functions. The methods used here are getSession(), which just gets the value of the JSF "session" variable, and the xmlEncode() and specialTextDecode() functions from my string utils. Additionally, it references "com.raidomatic.xml.*", which are quick wrapper classes I made to ease basic XML access, and which are also available on GitHub:

https://github.com/jesse-gallagher/Domino-One-Offs/tree/master/com/raidomatic/xml

Enhancing xe:dynamicViewPanel For My Own Purposes

Feb 9, 2012 8:45 AM

I think I have my view rendering problem licked. To recap, I've been working on a way to show views in XPages that met a couple requirements:

  1. Entirely dynamic. Since this will be for a combined reporting site that will show views from customized project databases of wildly varying needs, I couldn't make any assumptions about view layout, categorization, or content. It should pull as much display information from the view design as possible.
  2. Fast. Some of these views have thousands of rows, so it should be as fast as possible to load from the database and render for the browser.
  3. Pagination. Though I don't really like pagination for the average case (who wants to look at a 40-row view 30 rows at a time?), sending thousands of table rows to a browser causes a miserable experience even when that browser isn't IE.
  4. Full-text searching and other Domino goodies. I don't want it to be a pain to just be able to search through a view like you would in the Notes client.
  5. Support for "second tier" Domino view features. I use special text, column hide-when formulas, "show values in this column as links", Notes-style [<span>pass-through-HTML<span>], and color columns constantly.

Of these, 1 and 5 are the most important.

I originally wrote my own code to generate a big HTML blob for the view, which was perfect on points 1 and 5 (and looked great, since I converted categories to OneUI sections), but wasn't as hot on the other points. The Extension Library's xe:dynamicViewPanel control hits points 1 through 4 with aplomb, but misses the crucial point 5. For one crazy moment, I toyed with the idea of trying to get the output of the "legacy" HTML view renderer (which is perfect on all points) via client-side JavaScript or some other hack, but decided against it as being too rickety.

I considered changing my custom HTML code to instead create a bunch of Java objects to make pagination easier, but performance would be an issue - it's tough to beat built-in controls for raw speed. As anyone who has looked at the code for the mail template knows, IBM/Lotus cheats like crazy, and often the only thing you can do is to just go with what they've done and beat it into shape. Thus, the answer came to me: customizer beans for the ExtLib's dynamic view panel.

The dynamic view panel has a "customizerBean" property that takes a string class name of a bean to create to hook into a couple overridable methods and events:

  • ViewFactory getViewFactory() - this lets you override how the panel pulls in all of its information about the view.
  • boolean createColumns(FacesContext context, UIDynamicViewPanel panel, ViewFactory f) - this lets you override how the panel uses the gathered view information to generate the list of data table columns (or simply run code before that happens, if it returns false)
  • IControl createColumn(FacesContext context, UIDynamicViewPanel panel, int index, ColumnDef colDef) - similarly, this lets you change the way each individual column is created based on the ColumnDef (an object generated by your ViewFactory)
  • afterCreateColumn(FacesContext context, int index, ColumnDef colDef, IControl column) - this is an event fired after each column is generated, allowing you to customize the generated column without having to build it from scratch
  • afterCreateColumns(FacesContext context, UIDynamicPanel panel) - similarly, this lets you run code after the column creation is done

These events provided just the hooks I needed to get the dynamic panel to do what I wanted.

Special Text

To add special text support, I added an implementation of afterCreateColumn(...) that replaced the column's default ViewColumnConverter with my own subclass that overrides getValueAsString(...). With that, I can walk up the current component's ancestors to find the panel, find the individual ViewEntry's variable name from there, and use that to process the special text properly.

Column Hide-When Formulas

These are a bit trickier. The getViewFactory() hook lets me provide my own subclass of DefaultViewFactory, but there's a reason that the dynamic view panel doesn't support these formulas by default: they're not exposed by the NotesViewColumn class. The two ways I can think of to get at this information are the C API (or otherwise reading the design information from the view note directly) and DXL. The former is no doubt significantly faster, but DXL is much, much easier and is good enough for my needs at the moment. So I export the view as DXL and process its column nodes and their children. Then, for each with a hide-when formula, I evaluate the formula in the context of a new document in the project's database and use the result to set the column-hidden flag.

"Show values in this column as links"

This goes hand-in-hand with the hide-when formulas. The column nodes in the DXL have a showaslinks attribute that I can look for to set this flag.

Pass-through-HTML

Much like special text, once I have the custom converter attached to the column, I can break apart and reassemble text values on [< and >] and XML-encode the non-HTML bits. While I'm attaching the converter, I set the column's content type to "html", since I'll be handling the encoding.

Color Columns

These are... tricky, and I'm not yet proud of my fix. There are two problems: getting each cell to know about its color information from previous cells and then actually applying that color information. For the former problem, I dealt with it by adding an "activeColorColumn" property to the ColumnDef object and setting it to the programmatic name of the most recent color column before each column. The latter is the ugly part. You can't just set the column's style, since that will apply to the entire column and make a fine mess of everything. What you really want to do is assign it to the current <td> element, but I don't know a good way to do that. In the mean time, I've hacked around it by making a hidden <div> in each cell with the style tucked into a data-cell-style attribute. Then, I have some client JavaScript look for these divs and apply their styling to the parent table cell. It's not pretty, but it works for now.

If I get that last bit of code cleaned up to a point where I'm comfortable letting people see it, I'll post my example code. In the mean time, I'll count this whole thing as a win for only writing the code you need to and letting the platform take care of the rest.

Formatting View Content on the Web

Feb 1, 2012 10:06 AM

Tags: domino

My current work project involves displaying arbitrary views from various databases into a combined reporting site (written in XPages). This has presented me with two hurdles: pulling in the data completely and accurately and then figuring out a good way to format it.

The former problem is one of the few areas where it seems like "classic" Domino development has an edge: views render rapidly and, as long as you've configured the server to display lots of rows or add in pager controls, completely. And with the recent "enhanced HTML" property, they generally get tagged with some useful CSS classes. The default XPages xp:viewPanel control, by contrast, needs the column definitions at design time, which makes it tough to work with. You can sort of roll your own with nested xp:repeat controls, but you quickly run into problems with categories and other Notes-y features. Fortunately, the Extension Library, as usual, comes to the rescue: the  xe:dynamicViewPanel does pretty much what you want: you give it a view and it renders it out nicely. Using it, you don't have to worry about handling categories, you can add in standard pagers, and you can include all the standard filtering and searching options on the view. It's not perfect (yet), though: it doesn't handle fixed-value columns (like a formula of "hi"), use color-column information, follow column hide-when formulas (though it supports the static "hide this column" checkbox), do Notes-style bracketed pass-through-HTML, process special text, or honor "show values in this column as links." These are all understandable limitations, being as they are either rarely encountered or technically difficult. Unfortunately, I've run into all of them rapidly, so I don't know if I can use xe:dynamicViewPanel as-is. The best thing to do would be for me to roll up my sleeves, implement these features in it, and submit my changes, and maybe I'll do that down the line, but there'd be a lot of learning overhead before that point. What I've taken to in the mean time is a Java class to write out the view contents as HTML. I add the color-column info via inline style="..." attributes, look up the hide-whens by exporting the DXL and then running them through session.evaluate (read: not fast), and split and recombine the strings to handle pass-through-HTML. Unfortunately, I lose speed (because I'm presumably not as good at this as IBM's programmers) and the flexibility to easily use searching, filtering, and paging. The first two will be relatively easy to implement, but paging will be a problem, since the XPage just gets back a big string blob. I could return Java data structures, but then I'd run into problems with categories. I could write classes to dynamically add XSP elements to the page, but at that point I'd be best off cracking open the ExtLib source code anyway.

The other problem is that, regardless of how I render the view, it can be tough to actually display all the data well on a web page. For normal views, with a handful of columns, it's fine to just plop it right on the page and be done with it. However, it doesn't take much to make the table look like crap: headers that are way too long, large strings in the row data, or just having lots of rows all very quickly stretch or break the design. I think I'll try to handle the headers by adding some CSS text-overflow rules to keep them to a fixed width, which will also mean adding in a title="" attribute so the user can hover over them to see the full value. Still, not a big problem. Large data cell values are a bit tougher. Theoretically, I could do the same thing: have the data overflow and add a tooltip. That would work fine if it's a rare occurrence, but what if the whole point of the view is to be able to easily look at that data. I may try to pull out the row-height property from the view and use that to provide a max-height to each cell via inline CSS, so that way I could solve the problem on a view-by-view basis. I could consider doing the same with width as well, rather than going the default route of letting the content width determine the column width. Having a lot of rows can be partially solved with pagination (unless I'm going the "dump out a blob of HTML" route), but is that really a good user experience if the goal is to see all of the data? Paging on the web is pretty awkward, with the number of rows never exactly matching the height of the viewport, worrying about scrolling back to the top when going to the next page, and properly dealing with categories. Via CSS, I've made it so that the primary content area has its own scrollbars via absolute positioning and overflow: auto and that works pretty well for the most part, but I still run into trouble with views containing thousands of rows: with a lot of data, it simply takes a long time to download the HTML and render the table. There are a couple options provided by Dojo and the Extension Library that provide fixed-size grids to store data that I've tinkered around a bit, and maybe I'll settle on one of them, but in the mean time every solution has some serious problem that I quickly run into with my example data.

Getting My Feet Wet With Ruby and the Domino C API

Jan 26, 2012 4:07 PM

Tags: domino

My search for a useful way to use Domino as a database back-end for a Ruby front-end has continued and, more specifically, has continued to be difficult.

I gave a shot to the Java CORBA API. Initially, that went well: after grabbing NCSO.jar from my Designer installation, enabling DIIOP on the server, and setting up JRuby, I was able to connect to the server using the host name, canonical user name, and password. Great! However, once I exported my app's Java model classes (modified to use NotesFactory instead of the JSF runtime), I immediately ran into a problem: NotesView.resortView() is not implemented in CORBA. Guh. Theoretically, I could rework the app to use a different view for every index, but it's a canary in the coal mine: no doubt plenty of other new things (FTSearchSorted, for example) have been left unimplemented as well. Plus, this is Domino, so "new" probably refers to anything added after, say, R5.

So that's out, at least for now. That leaves me with the C API. Oh, the C API... so much promise, such a PITA to use. I haven't used C itself much since college, and my experience with the C API is limited to progress bars in the Notes client and, more usefully, my authentication filter for my server. However, I'll be programming in Ruby, not C directly, so it won't be so bad, right?

There are three main ways to work with C libraries in Ruby, I've learned: writing a Ruby module, DL, and FFI. In the first case, I'd basically have to write my API in C and then just use those objects from Ruby. That would be the best performance, but it's just asking for a nightmare. DL is programmed in pure Ruby, but it's so sketchy that just loading the library causes your program to emit a warning telling you you probably shouldn't be using it. FFI, though, is pretty great. Much like DL (or declaring external functions in LotusScript), you declare which functions you want to import, along with their signatures, and then it more or less just works.

That "more or less" is, naturally, tricky. I ran into tremendous trouble just getting it to load up libnotes.so, but most of that was due to using a 64-bit OS and a 32-bit Domino installation. I may make a post about it some time in case anyone else is crazy enough to try the same thing, but the upshot is that I ended up compiling my own 32-bit Ruby environment, which started to work.

Since then, I've been wrestling with the inherent traits and idioms of C (using string buffer arguments to return values instead of just returning a "string", for example) and the specifics of Notes (needing to be pointed to the executable directory as well as needing an ID file, an INI file, and a names.nsf to find the servers). After thrashing around for a while, though, I have it working to the point where I can point the initialization routine to an arbitrary INI file and it can pick up on everything it needs from there (I'm using a password-less ID file - I don't know if you can automate passwords).

Now that I have my foot in the door, it's "just" a matter of learning the entire API. I'm looking forward to that part, though - I've demonstrated that I can open a database on a server and get information out of it, so it seems like I have the right environment. If all goes well, I should be able to create a proper Ruby API for Domino, and a very fast one at that.

 

For reference, here's the script I've written. Don't expect clean code, reusability, comments, or proper methodology - this file is the result of random mauling during the course of development and just barely works to demonstrate the idea. It DOES demonstrate it, though, and works to ping a (hard-coded) server and get some info about its names database.

test.rb

A Couple Handy Domino Java String Utils

Jan 25, 2012 11:08 AM

As most developers probably do, I have a grab bag of "utility" functions/methods I use across various projects, and I figured I'd post a few of the handier ones.

The first is a basic XML-encoding function that just takes a string and returns something suitable for putting into an XML or HTML file. I've been too lazy to figure out if the standard Java library has an equivalent that doesn't involve creating an actual XML document in memory, so I just took the one I use for LotusScript and ported it over. It passes standard letter and number characters through as-is and then just encodes all others as Unicode entities. The string size increases accordingly, but it's presumably fast and it gets the job done, even if you have oddball characters.

The next set are just quick-and-dirty Java equivalents to the StrLeft, StrRight, etc. functions from LotusScript. They do more or less the same thing as in LS and make it so you don't have to worry about string indexes all the time.

The last is potentially the most useful, since it's the most esoteric: a special-text decoder. I've had a couple occasions where I want to spit out the contents of a view for the web, but also want to preserve things like child counts on categories. Fortunately, special text is stored in the view column in a workable format: a special non-alphanumeric character, a single letter indicating which function it corresponds to, a number representing the parameter count for the function, and then a delimited list of those parameters with character counts. Using that, I wrote a routine to process all of the special text functions I could think of other than @IsExpandable (since that's meant for the Notes UI). It even reproduces the weirdo behavior you get when you pass a multi-character string to @DocLevel.

StringUtils.java

Confound It; That's Two APIs Down

Jan 22, 2012 11:02 PM

Tags: domino

In the interests of getting crap done, as soon as I was finished with my previous post, I fired up TextMate and a couple Terminal windows to start writing a Ruby wrapper for the DAS API.

It started out great! Wanting to avoid the minor hassle I ran into before with Ruby's built-in Net::HTTP library, I did a quick search for Ruby HTTP/REST libraries and picked one that worked well, named HTTParty. Before long, I had the rudimentary elements of a Notes API working, with classes to represent the server connection, NotesDatabase, and NotesView.

I was about to start working on the NotesViewEntry class when I decided I'll need a way to page through the view, since the call to the view "collection" (wisely) doesn't return all of the data at once. The API developers cleverly store this information in an HTTP header:

Content-Range: items 0-9/35

Perfect! But, um... I was getting the list of available forums as Anonymous, which can only see two, not all 35. Sure enough, the results only contained the two Reader-visible entries (phew), but that left a showstopping problem: I have fairly frequent need for the count of entries, and I don't relish the notion of having to fetch all of the data to get that. Crap.

So DAS is out for now. I remembered, though, about ?ReadViewEntries, an older and thus clearly more mature data-access API. I could probably do everything view-related I need with that and then use DAS to get any non-view document data. So I hit the view with ?ReadViewEntries&OutputFormat=JSON (the results are the same with the XML version) and, lo and behold, the top-level object fields (with data snipped) revealed the same lack of Reader knowledge:

{ "@timestamp": "20120123T034916,85Z", "@toplevelentries": "35" }

Crap!

My quest to treat Domino like a standalone database is off to a rocky start. I can think of a few remaining options:

  1. COM: I don't deploy on Windows, so... that's out.
  2. C API: seems like overkill and just asking for all sorts of new bugs.
  3. Web Services: MIGHT work, but I can't imagine the performance would be anywhere near acceptable.
  4. Some sort of crazy thing with servlets: seems too crazy.
  5. JRuby and the Domino Java CORBA API.

I think the last is the most plausible. My forum app already has Java classes in between the XPages front-end and the Domino database, so I could possibly take those wholesale and just change the methods I use to get the session and current database to instead initialize a CORBA session. I'm a bit nervous about speed and running across some of the various "not implemented in CORBA" landmines hidden across the API, but it might just work. If it does, my next task will be deciding on a web server and a Ruby web framework that's not terribly SQL-centric.

Keychain DB: Very Rough First Script

Jan 15, 2012 4:20 PM

Tags: domino

I've had a chance to start on my Keychain project from last week, enough to put together a thoroughly rough and unmaintainable script to do the uploading. It has all kinds of horrible properties: it doesn't do the Keychain dump itself (instead reading from a hard-coded file containing a keychain dump from the "security" tool), it doesn't abstract away any of the Domino DAS access, it doesn't check for existing versions of the items, it doesn't handle field data types properly, and it even writes to the filesystem. But hey, it's a start:

keychain-upload.rb (requires the json rubygem and uses auth information from ~/.netrc)

In spite of all its faults, it DOES push the data up to the database, and that's something. It stores the keychain file name in a field named "keychain", the "class" in one called "entry_class", the data in "data", and the rest of the arbitrary fields from the item in fields prefixed with "attr_". Once it was in the database, I set up a view called "Passwords" with the following formula:

SELECT Form="Item" & entry_class="genp" & attr_type != "\"note\""

The extra quotes around the type field's value are an artifact of the non-existent type handling in the original script - it just stores the values exactly as they appear in the dump file, rather than converting them to null, numbers, or strings.

It's not pretty, but I now have a way to view my Keychain entries on the web. I'll probably give a shot to doing this the "right" way via a Cocoa app and structured classes down the line, but for now it'll already be useful to me.

A New Personal Project With Keychain and DAS

Jan 11, 2012 7:18 PM

Tags: domino

With Apple's transition to iCloud, they're getting rid of Keychain sync. This is too bad, since that was one of the MobileMe features I actually used, loved, and never had problems with. Fortunately, all is not lost: worse comes to worse, I can pick up a copy of 1Password, which has the advantage over MobileMe of being cross-platform.

But I'm a programmer, right? Why buy something - especially for $50+ - when I can just write something myself? My needs at the moment are fairly simple - I only REALLY want a way to back up my Keychain and view it on the web, for when I'm in Windows or on my phone and want to check a password. I only use the one Mac currently, so I don't need proper sync, and I don't need to pull new passwords back down into the Keychain. Maybe down the line, but one step at a time.

The Keychain itself takes the form of a file housing a collection of records with a body of essentially arbitrary data (usually a string, but it can be different for certificates or notes), with a composite key consisting of, I think, the type, server, account, and maybe some other fields. The only thing that would make this a better test case for Domino 8.5.3's shiny new Data Access Services would be a 32-digit hexadecimal unique identifier, but it's pretty darn close as it is.

The main problem I have to getting starting is actually getting at the data. I think there are two main ways: the C-based security/Keychain API or using the "security" command-line tool, which is a more-or-less friendly wrapper for much of the same functionality. Being the lazy type that I am that only deals with C when I have to, I'm starting with the "security" tool. It has a handy "dump-keychain" command, so I can type something like "security dump-keychain login.keychain" and get a formatted list of all entries in that keychain, minus the "data" field. I can add the "-d" switch after "dump-keychain" to include that field, but that comes with a big caveat: you have to click an "Allow" button for every single item. Given that I have about 1,100 items, I'd rather avoid this fate. I've tried running it through sudo, but to no avail. I've tried fiddling with the "authorize" command in the tool, but also with no luck - I'm not sure if I'm just doing it wrong or if it's related to something else, which may or may not be what I want. I could use the "-r" switch instead of "-d" to get the encrypted data, which would satisfy my backup desire, but not my "check it from the web" desire. Absolute worst case, I could put on a movie or podcast and click "Always Allow" a thousand times.

However, once I get past that hurdle - whether it be via the C API or a lot of clicking - the rest should be easy. I should be able to create a basic database, grant DAS access to it, create a view sorted by the various components of the items' composite keys, and do a basic "view lookup, then update or create the doc" routine, pushing up the field values pretty much as-is. That should act as a perfect basic-case project for trying DAS out while also solving an actual problem I have.

Pondering RSS Syncing

Nov 23, 2011 1:10 PM

Tags: domino

I was listening to the latest episode of Build and Analyze on the way home from work yesterday and, as I am wont to do, I started yelling at my iPhone when they started talking about Google Reader and the difficulty of syncing. Admittedly, at the end, they got to the fact that, even if you could do it technically, it'd be tough to make money off of providing an RSS sync server. That part is fair enough, but I still can't let the technical difficulties stand, and I've been thinking more about how it would be done in Domino.

In the basic form, the problem in question is pretty much exactly what NSF and the Notes/Domino relationship is designed to do: seamless replication, deletion stubs, unread marks, and so forth. In fact, RSS syncing is a better fit for the model than mail, since mail required adding all kinds of extra (but useful) functionality, while RSS syncing would just be data and an agent to fetch the feeds periodically.

The way I figure it, there would only be a couple technical hurdles, both related to scaling: storing large volumes of data and fetching new feed content periodically.

Storing large volumes of data might not be too bad. There are a couple ways you could do it. One would be to store the user's list of subscribed feeds and "read" stubs in one database per user, and then store the feeds and feed content in another database (or databases), and do all data access via agents or web services that would pull the data from each distinct location. Another way could be to store the feeds and entries in each user's database, keeping the feed content as attached HTML documents and letting DAOS handle efficient storage. The latter route would let you take advantage of the Domino Data Service and read marks (which DAS conveniently supports).

Updating the feeds would be rough for a single server, but the job could be farmed out to many clusters in a server. You could write agents that would determine the server they're on and, based on, say, its name, pick a slice of the feeds to check, so if you have 10 servers, each would update 10% of the feeds.

I'm sure there'd be other roadblocks during actual implementation (it IS Domino, after all), but I think that'd be basically all you'd need on the server side. The client would be a little tougher, since you couldn't just use NSF and selective replication, but that wouldn't be terribly difficult to handle.

It's too bad it's likely not profitable, between licensing, hosting, and bandwidth costs - it'd be a fun project to try out.

That Counts as Progress

Nov 22, 2011 4:28 PM

Tags: domino xpages

A while back, I described the problem I'm having in my guild-forums XPages app, which is that it very easily gets its environment out of whack, to the point where changing any design or data note from outside the XPages environment caused an "X is incompatible with X" ClastCastException. This improved gradually over time. At some point in 8.5.2, I started being able to modify data documents again without the problem. When 8.5.3 came out, it improved again: I can now replicate over changes from my development server to the production one without having to bounce HTTP on the production one. At that point, it became much less of a hassle - sure, I still had to re-save a Java class file every time I modified an XPage, but that was only in development, so I could deal.

However, I think I've found the proper solution. In 8.5.3 (I think), IBM added a custom editor for the xsp.properties file (which you can reach using the "Package Explorer" Eclipse view, in whatever.nsf/WebContent/WEB-INF). Many of the options are duplicates of what you see in the normal "Application Properties" editor (since they edit the same properties), but there are some nifty additional goodies. I won't go into them all, but I urge you to take a look. The important one here is in the "Timeouts" section of the "General" tab: "Refresh entire application when design changes" . That sounded perfect and, lo and behold, it seems to do what I want: once I turned that on, re-saving an XPage stopped causing the ClassCastException. I haven't given it a proper testing, so it might not be everything I'm looking for, but I'm thrilled at the initial results. Having to re-save a Java file for each change wasn't a HUGE problem, but it was a niggling one, and not having to jump through the hoop helps make the whole environment seem less rickety.

That's Weird

Nov 9, 2011 9:35 AM

Tags: xpages domino

Yesterday, I started working on a small sidebar widget app using an XPage, after finding out that XPages can now (as of 8.5.3) be used in the Notes sidebar in the same way that Forms could before. It's quite a simple page, very Twitter-like: one text input field and then a list of posts. However, even though it's very simple, I ran into two annoying bugs quickly.

The first of them is a Schrödinbug. I set up some code in the "onClientLoad" event to start a setInterval to do a partialRefreshGet on the list of posts, so it would keep itself up to date:

<xp:eventHandler event="onClientLoad" submit="false"> <xp:this.script><![CDATA[ setInterval(function() { XSP.partialRefreshGet("#{id:messagesPanel}") }, 2000) ]]></xp:this.script> </xp:eventHandler>

Simple, right? But I started getting errors in the status bar, along the lines of somethingorother not being defined. That's... odd. So I swapped over to Safari, figuring it'd be some XPiNC-specific thing, but I started getting similar (albeit more specific) errors there, along the lines of:

TypeError: 'undefined' is not an object (evaluating '_166.formId')

Huh. So I tracked it down and the problem was in the code called by partialRefreshGet, where "_166" is the name given to the second (optional) parameter. It makes sense when seeing it - though the code is supposed to fail over when it doesn't work, I can see why the browser would throw up its hands when you try to get a property of an un-provided parameter. But this has worked before! Just looking around on the web, you run into plenty of examples that leave out the second parameter. But sure enough, when I changed the line to XSP.partialRefreshGet("#{id:messagesPanel}", {}), it started working great. I can only think that this is either some 8.5.3-specific bug or some weird thing I managed to do in my code... but there's not even really enough code to mess up.

The second bug is still annoying, and it's somewhere in between a bug and a security feature of Firefox/Gecko. Basically, while you can call the .click() method on a button in client-side JavaScript in Gecko, it doesn't behave exactly like clicking on the button. Specifically, I have a CSS-hidden button that executes the actual action of creating the message document from the text you type in, and I want it to happen when you hit enter. The button itself works great - if I have it show up, I can type, click, and it executes the action and partial-refreshes the list of posts. However, if I hit enter, which uses the .click() method, I get an error about not being able to refresh that part of the page, but then it forces a full-page refresh and still works. So it's KIND OF clicking it, but not quite.

I'm not sure what to do about this one. I looked up a couple things online about trying to emulate the actual click event, but with no better results. I could do a REST service on the page, but I don't want to have to roll out the Extension Library to everyone in the company. Maybe I'll look into sending along the data in a XSP.partialRefreshPost call and eliminate the button entirely. We shall see.

Trying To Escape From Designer

Nov 8, 2011 11:14 AM

Though I've grown to more or less enjoy writing Domino applications, I always feel like this is in spite of the tools, namely Designer. In a lot of ways, Designer has improved significantly over the last couple versions: as long as you ignore the speed, the Eclipse-ified Java and LotusScript editors are miles ahead of the antiquated previous ones, and it's handy to be able to switch to the Java perspective. However, so much else makes it a drag:

  • It's a Windows app. I use a Mac, so there's simply a big hurdle to using Designer. Parallels has smoothed the process greatly - running Notes in Coherence mode means I can use my preferred OS while still getting work done. However, it means that it's a big to-do whenever I want to make any tiny change in the code. For day-to-day work stuff, I can get a lot done in the Mac Notes client, since I put in a lot of work to make managing client web sites doable without going to the design side, and the Mac client still lets you edit views and agents. However, there's no XPages editor, so I can't use that for serious work.
  • It has a mind of its own. For some reason, a new clean install of Designer I made the other day got it into its head that opening any database should involve recompiling every XPage and Java class. Sometimes, it turns off "Build Automatically" for no reason, and then turning that back on also requires a full recompilation. Sometimes, it just holds up all user actions while it does... something for five minutes. What's it doing? Beats me.
  • How many times have I seen this window? A billion?
    Removing a database from the project list has about a 30% chance of causing a crash and quitting Designer has about a 50% chance. Sometimes, saving a form will do it. Sometimes, walking away from the computer to get a drink is enough. And every time it crashes, I have to dismiss the dialogs and check the task manager to get rid of any residual processes, such as an instance of nsd pegging a processor, then relaunch Notes and get back to the environment I had, which is not a speedy process.
  • I'm not that crazy about Eclipse. DDE is better than previous Designers, yes, but Eclipse is still a giant beast with the same sense of style and simplicity as the monstrous Java language that spawned it. On the plus side, the blue look that IBM came up with is actually rather attractive compared to the standard Eclipse UI, but little quality-of-life things are a drag. For example, how do you specify how many spaces a tab should take up visually? It's in a couple places in the preferences and you have to set them all, including buried inside a weird sub-preferences dialog for messing with your Java formatter. And how about changing your code syntax coloring? You have to go to a dozen places, one for each syntax type. Compare to a text editor like TextMate, where you can swap between packaged color schemes with a drop-down.

That's enough for the rant. So what is there to do about it? I give this problem a thought from time to time, and I don't think there's really a great option, but there are some places to start. The real key to any alternative scheme is DXL - using that, you can (more or less) view and modify design elements freely. I've toyed with this notion before - maybe a web UI that lets me pick the DB and design element I want so I can tweak the DXL manually for when I want to make a quick change but don't already have Windows or Designer open. It would mostly work, but it would take a lot of work to make it practical.

There's a big sticking point, too: XPages. If you export an XPage to DXL, you can see that the exporter basically punts on it - it's exported as a generic "note" type with Base64-encoded binary fields. I haven't run the field data through a decoder, so maybe one of them is the XML "source" of the page, and maybe it could be made to work, but that just raises more questions. Would that require updating the other binary fields in some way? Would importing it with a DXL importer cause it to generate and compile the Java representations? What about generic Java classes? Would those work with DXL and would they be compiled?

I'm starting to get an idea of what would be ideal and almost practical, though. One could write a WebDAV server (possibly with a complex servlet, a small web server to the side, or other trickery) that represents the design elements as editable files in a folder structure similar to the Java perspective view of the database. Traditional files and image resources could be edited as-is (which I think the built-in WebDAV server does), but design elements could be represented just as DXL and then re-imported when modified. Even if it doesn't support XPages, such a scheme might have a lot of promise and wouldn't be reliant on Designer or any other IDE.

If I ever get brave enough to delve into WebDAV or frustrated enough with Designer, I might just look into it myself.

So Here's Why I Hate LotusScript

Oct 29, 2011 1:44 PM

Tags: domino

For the most part, writing agents in LotusScript is the best way to go (at least when it can't be done in formula language), mostly for smoothness of interaction with the built-in libraries (no .recycle()) and because it's less prone to running into memory problems when other agents go wonky than Java agents are. That doesn't mean I have to like it, though.

If there's one thing that drives me nuts about LotusScript more than any other aspect, it's its handling of arrays. This came to the fore with one of my recent projects, which involves spitting out the contents of a view, which in turn entails lots of use of entry.ColumnValues. My first, quick-and-dirty draft ran into an early performance issue, which is that every call of .ColumnValues on a NotesViewEntry seems to be as expensive as the first, meaning that the class doesn't do any internal cacheing and has to re-fetch it every time. Ugh, fine - I'll just assign the value to a new variable at the start:

Dim colValues as Variant colValues = entry.ColumnValues

Not too bad - two extra lines at the start of a loop is a small price to pay for a significant speed improvement. Unfortunately, it doesn't work - it throws a Type Mismatch error at runtime. After doing a TypeName() on entry.ColumnValues, I saw that it considers it "VARIANT( )", an array of Variants, which makes sense. It's a bit weird, since I've stored arrays in Variants before, but whatever - with a quick code adjustment, we're off to the races:

Dim colValues() as Variant colValues = entry.ColumnValues

Great! Now hit Ctrl-S and... compiler error. You're not allowed to assign to the entirety of an array like that. Argh! So I guess I'm going to have to make my new array manually and loop through the original to copy each entry over individually. If you're familiar with another scripting language, that probably sounds like a simple task, but LotusScript's array annoyances continue. Because this is like BASIC, you can't just do "colValues.add(something)" - you have to ReDim the array to the right size. Here's the code I ended up with:

Dim columnValues() As Variant ReDim columnValues(-1 To -1) ForAll columnValue In entry.ColumnValues If UBound(columnValues) = -1 Then ReDim columnValues(0 To 0) Else ReDim Preserve columnValues(0 To UBound(columnValues)+1) End If columnValues(UBound(columnValues)) = columnValue End ForAll

Before you look at that "-1 to -1" crap and deem me insane, hear me out. Though the NotesViewEntry doesn't cache its property value, the ForAll loop does, meaning that, according to the profiler, ColumnValues is only called once for each view entry, which is about as efficient as it gets. All that extra crap about ReDim'ing the array over and over instead of just once is essentially "free" compared to the expense of the product object call, so it ends up being completely worth it.

Sync

Oct 25, 2011 9:52 PM

So there's a new round of talk lately about syncing and the trouble involved, thanks to some changes in Google Reader's behavior and the desire to find a new safe haven for RSS syncing. The best example is, unsurprisingly, from Brent Simmons:

Google Reader and Mac/iOS RSS readers that sync

However, the whole time I was reading this article, my brain kept yelling at me, louder and louder as time passed:

This is Lotus Notes! The system you're describing is Lotus Notes! It does syncing and deletion stubs and read marks! IT'S LOTUS NOTES!

This kind of thing would indeed be really easy in Notes/Domino, particularly if you were actually using the Notes client (though it wouldn't be much to look at). Subsets of data, managing deleted elements, timed refreshes from the source, storing each feed entry as its own entity, and offline access that can have its changed synced back to the master are all things that Notes has done since its conception - the only problem is that it's so ugly and arcane that mass-market appeal is nigh-impossible.

Nonetheless, it got me thinking about the viability of using Domino as a syncing server for this. You wouldn't be able to use NSF files in your RSS clients, which would make the job a bit tougher, but the new "XWork" licensing model would fit into this nicely. Scalability would be a serious concern, but the simple nature of the data would keep view updates quick, and it'd just be a bit of cleverness in the database layout to direct users to the correct place. Toss a couple clustered servers in there and you should have some good load balancing, too. The Domino Data Services API might be enough to handle data access from the client, but, if it's not, a couple simple agents would do it.

I'm sort of tempted to try hashing something out.

The Domino Data Service

Oct 5, 2011 10:22 AM

Tags: domino

Though I don't have a use for it currently, I can't help but get kind of excited about the Domino Data Services in 8.5.3 and the Extension Library. If you're writing a normal Domino application - using either legacy elements or XPages - you probably won't have terribly much use for it.

However, the really cool aspect of it is that it significantly smooths the process of using Domino as a backing data store for another front end written in PHP, Ruby, or anything else. This has always been sort of possible - you could use the Java API or a combination of ?ReadViewEntries, ?CreateDocument, and ?SaveDocument URL commands to access Domino data without actually being in Domino, but it wasn't exactly a smooth process. With the Data Services, now Domino is very similar to, say, CouchDB, but with reader fields and impenetrable licensing terms for non-vendors.

One nice little side effect of the fact that it uses the HTTP stack is that DSAPI modules work. When I was testing around, I was able to get a list of available forums as Anonymous, resulting in only the two visible ones. When I included my user authentication filter cookie, it started showing me the rest of the forums that the user could access, exactly like you'd want. While you could presumably just pass the username and password in each request using normal HTTP authentication, it's pretty cool that any alternate methods like this work as well.

It's all pretty exciting, and I'm itching to find a use for it.

My next two favorite features of 8.5.3

Oct 5, 2011 8:07 AM

Tags: domino

Since 8.5.3 has been out for about 24 hours now, I naturally rolled it out on both my development and production servers. Fortunately, my irresponsibility was greatly rewarded: the largest problems I've had so far were a change in the way Java classes are accessed in JavaScript (I could no longer just call methods on non-public classes defined in the same file as a public one, so I had to split them out into their own files... which is what you're supposed to do anyway) and a minor CSS change where the top borders of my Dojo tabbed tables are now back to a light grey color, so I need to find the new CSS rule to change them back to brown.

I'm rather happy so far about two minor things in particular: CSS/JavaScript aggregation and OSGi auto-loading.

The CSS/JavaScript aggregation is almost a freebie: once you have Designer 8.5.3, you get a new option in the database properties sheet to turn this on and then 8.5.3 servers will happily obey it. I immediately noticed a decent load-speed increase of about 1/3 and one non-technical guild member said that the odd problem of dog-slowness that they (and not other people) had has been fixed. My favorite aspect of this is that it's a smart feature: due to the way you define Dojo modules in an XPage as <xp:dojoModule/> elements and not just text on a page like normal HTML, Domino knows ahead of time what you're using and can thus feel free to optimize it in transparent ways like this. It feels good seeing the same code go from one form to a more efficient one just by virtue of done the "right thing" when writing it to begin with.

The OSGi plugin auto-loading was mentioned briefly on Dec's Dom Blog back in June and I hadn't seen much reference to it since, so I was afraid it wouldn't necessarily make it in. Fortunately, it has: I created a new Update Site with the template from an 8.5.3 server, imported the latest Extension Library, ran the "Sign All" agent, and set the notes.ini parameter to look there. And lo and behold: it properly loads up the extensions from the NSF, so I was finally able to delete the filesystem versions that were previously necessary. This makes managing the Extension Library much smoother and it's one less potential gotcha when I upgrade my dev server first and then want to deploy it - since the Update Site has a replica on both servers, the upgrade is handled with the normal replication process and I don't have to remember to copy any files over from server to server. And the fact that it's an NSF theoretically gives you all kinds of other, more complicated deployment options, like server-based Reader field control or partial replication to control which servers see which plugins if you're so inclined. Very cool - I approve, IBM.

My Favorite Minor Feature in 8.5.3

Sep 28, 2011 6:23 PM

Tags: domino

I don't have access to the beta versions of new Notes/Domino versions, so I haven't been able to tinker around with all the cool new things that are slated to appear in 8.5.3, but that hasn't stopped me from getting pretty excited about some of them. The big-ticket items are clear: the new Domino Data Services and relational database access (through the Extension Library, which may as well be standard) could make practical very different ways of using Domino either as a standalone data source with a different front end or as a standalone front end with a relational data source. While both types of setups are theoretically possible now, they're such a hassle that they're not worth it - but, with 8.5.3, they're almost top-tier choices for system architectures.

However, since I don't plan to rewrite my entire architecture, those new features probably won't affect my day-to-day life for a while yet. What has me most excited in a practical sense is much more lowly: being able to do a full-text search with sorted results. I've found that one of the big bottlenecks in my guild-forums app is the sheer size of the views, particularly the Posts one. I used to stuff pretty much all of the summary data into the view, but then I found that removing the non-sorted columns sped up responsiveness dramatically. That whetted my appetite for clearing out unneeded sorted columns - since each sorted column contains a full view index, having even a handful can increase the total index size dramatically. Since it appears that FTSearch's performance is almost (but not quite) as good as getting all entries by a key, I'll be able to remove the rarely-used sorted columns, speeding up all the common operations in exchange for a very minor hit in the rare case. Plus, it'll just feel good to put Domino's searching capabilities to proper use.

Getting Domino LDAP to Work for Authentication

Aug 25, 2011 4:23 PM

Tags: domino

Recently, I've been toying with the idea of setting up a couple extra services on my guild's Domino server - voice chat, non-Sametime chat, what have you - and I figured I should give a shot to LDAP authentication with the Domino directory for these. However, this is something I've never done, and the documentation is a little rough - most LDAP info on the web refers to non-Domino servers, while most Domino-specific information was written in about 1996.

I'll leave out the depressing details of the various things I tried in my quest to get LDAP working as an authentication mechanism for my Linux server (as a relatively simple test case) and point you instead to this dead-but-still-archived page: http://web.archive.org/web/20040614140723/http://www.dominux.co.uk/ldap.html. The key information on that page is the list of fields that you have to add to your user documents to use them for this purpose. During my harried testing, all /var/log/auth.log was telling me was "Invalid credentials", but what it really meant was that the user account it found didn't have the right attributes. Thanks, Linux!

Freaking Reader Fields

Jul 14, 2011 3:02 PM

Tags: domino

Periodically (read: every day), I wonder about switching from Domino to a SQL server for data storage on my guild web site. The primary reason for this is speed: I'm doing primarily relational things, so I've had to wrangle Domino quite a bit to do this with any amount of speed and code cleanliness. Additionally, while most of my documents are entirely distinct from each other, I had to make concessions here and there, such as storing the latest Post date in Topic documents so I can sort them that way, and, each time I have to do that, there's another little bit of code maintenance and clustering-unsafety.

However, my ideas always come to a screeching halt when I remember Reader fields. They're simply too good, and the replacements I've found on the open source SQL databases have been, to put it kindly, lacking in comparison. They generally involve having some access level field or, best case, a multi-value field of names that are allowed to see the document, and then making sure that all of your queries or views honor that. Every method has some severe downside, ranging from inflexibility (access level) to nightmarish piles of code everywhere (multi-value name/group/role fields). Everywhere I accessed the database, I'd have to worry about security and document access, bloating up the code and just asking for data-leak bugs.

Domino, for all of its faults, makes this something you just don't have to worry about. If you have a Reader field, you can toss names, groups, and roles in there with impunity, and the server will handle the rest like you'd want. You don't have to do your own directory lookups, security checks, or nested queries. If the current user isn't on the list, the document may as well not exist. Even if the user had the UNID of the document and designer access to the database, it'd be beyond their reach. This is enormously comforting. And even though it's just a guild web site and not a giant corporate database, I'd still rather deal with a bit of tricky code for performance than the headaches and drama involved with people seeing what they're not allowed to see.

So, until I either get entirely fed up with Designer or I find an equivalent to Reader fields in a free SQL server, I'll be sticking with Domino.

Starting work on a DSAPI filter for Domino 8.5.2 on 64-bit Ubuntu

Apr 14, 2011 4:33 PM

Tags: domino

For my main project, which contains a forum, one of the problems I ran into was Domino's session handling. Namely, it's designed such that HTTP user sessions last for the duration of the browser session and time out after 30 minutes. That's fine for, say, a corporate app, where you don't want a client logged in indefinitely. However, having to re-log-in to a forum every time you visit it would be a hassle.

The session timeout is easy to fix - you can just up the timeout period in the web site config. The cookie took a little more trickery, but wasn't too bad either. I set up some code in the beforePageLoad event to look for a DomAuthSessId cookie and, if present, create a persistent cookie. The server can't tell the difference between the session and persistent cookies, so this works.

However, it's sort of ugly. While I absolutely don't want to have people re-log-in every time they restart their browser, I also don't really want tons of session-scoped managed beans floating around for months, and this kind of trick defeats any use of the Internet Users list in Domino Administrator (a minor quibble, but a quibble nonetheless). Ideally, I'd be able to automatically log people in on subsequent visits, but not have to actually maintain server sessions for them. From all I've read, this calls for a DSAPI filter. Since that involves C/C++, there's bound to be a lot of setup and tribulation.

First off, I had to grab the Notes/Domino C API toolkit. Fortunately, Google led me directly to the IBM download page and, after re-confirming that I don't want them to send me email, I was given a download link for a quaintly compress'ed file and some installation instructions. I dropped the libraries into /opt/ibm/lotus/notesapi and set up a symlink at /opt/lotus/notesapi just in case. The samples in the toolkit do indeed include a DSAPI filter, and I found another one doing something very much like what I want here:

http://www-10.lotus.com/ldd/46dom.nsf/55c38d716d632d9b8525689b005ba1c0/dd8cc7c9ad12887a85256bca003476bf?OpenDocument

To keep things simple, I started with that code and commented/deleted everything that actually does something - I just wanted the simplest possible filter to see if it works at all. All it does is announce that it's alive and then proceed to not handle any requests:

dsapilogin.c

Now came the issues. First off, the DSAPI example in the toolkit only came with a handful of Makefiles, and none for Linux. The Solaris one was close, so I borrowed and modified that one with some tips from http://www-10.lotus.com/ldd/nd6forum.nsf/55c38d716d632d9b8525689b005ba1c0/fbd4bb196ce4146685256df10038a0b9?OpenDocument . The most vital thing was that, since I'm running in a 64-bit environment but using a 32-bit Domino server, I had to go out of my way to compile a 32-bit library - this is done by adding "-m32" to the CCOPTS and (maybe required) LIBS lines. The Makefile I ended up using is:

Makefile

Along the lines of the "-m32" flag, I had to install some additional development libraries to support 32-bit compiling. In my travails, I ended up installing all sorts of apt packages before it worked, so it's possible that only the last (or another, non-C++ library) is necessary:

g++
lib32stdc++6
libc6-dev-i386
g++-multilib

Unfortunately, even when you get the library building properly, it's tough to get it to load into Domino, and it's not very helpful as to why - all it says is "Failed to load DSAPI filter: " and then the file name... and it gives the same error if you point it to a non-existent file, so there's no way of knowing WHY it failed to load it. It's all trial and error (unless you actually know what you're doing, presumably).

Once all that was in line, though, I could build the library with make, copy it to /opt/ibm/lotus/notes/latest/linux, add "libdsapilogin.so" to the Configuration tab of my Web Site document in the Directory, and restart HTTP. When HTTP is loading, it will print out "HTTP Server: DSAPI DSAPIDLL Loaded successfully" (or whatever message you put in there), so you know it's working.

Now comes the task of actually writing the filter. Considering that I haven't written any C or C++ since college, I expect lots of fun with buffer overflows and mysterious crashes. Exciting!

Installing Domino 8.5.2FP2 on an Ubuntu 10.10 server

Apr 5, 2011 7:19 PM

Tags: domino

Because I am insane, I decided to install Domino on my newly-minted 768 slice from Slicehost. That's just above the 512MB RAM minimum that Domino demands and half the recommended 1.5GB, so we'll see how it goes.

The Linux installation I went with for my slice was Ubuntu 10.10 64-bit, which firmly puts it into the realm of unsupported, as far as Domino is concerned. Fortunately, three pages I found made the process relatively painless:

  1. http://www.collaborationmatters.com/blog/cmblog.nsf/dx/installing-lotus-domino-8.5.2-on-ubuntu-server?opendocument&comments
  2. http://www.danilodellaquila.com/blog/how-to-install-lotus-domino-8.5-on-ubuntu-part-ii
  3. http://stackoverflow.com/questions/3747789/how-to-install-sun-jdk-on-ubuntu-10-10-maverick-meerkat

The first wrinkle you run across is that Domino adores Java, but stock Linux installs do not. Fortunately, link #3 there explains how to cleanly install Java on your server. Namely, add this line to /etc/apt/sources.list:

deb http://archive.canonical.com/ubuntu maverick partner

Once you do that, you can run apt-get install sun-java6-jre, wince at the huge number of dependencies it requires, and let it do its thing.

After that, you're almost ready to start installing. If you're running on a 64-bit installation, you'll need the standard 32-bit libraries, since I don't think there's yet a 64-bit Domino server for any-old-Linux. That one's easy, though: apt-get install ia32-libs and you're all set.

The Domino installation proper was relatively straightforward. I created a "notes" user and associated group beforehand, picked and created some directories for the program files and data owned by that user, and went through its command-line wizard via ./install -console . For the final step, I told it to expect a remote installation and fired up the Remote Server Setup tool on Windows, which worked flawlessly.

The next step was the FP2 installer. As is the case with the Windows version, the installer for the fix packs is totally different than the normal installer for some reason. It initially complained about the size of my Terminal window and, rather than trying to convince it that my GUI terminal was plenty good enough for its installer, I took its advice and did export LOTUS_NOROWCOLCHECK=1 . After that, it complained about needing to know where the data directory was, but provided a similar instruction for setting that variable. Once that was in order, the installation went smoothly.

The next tough part was how to get Domino to run at startup. I've been familiar from time to time with various *nixes and their service setups, but it seems to change every couple of years, so I wasn't sure where to go with this. Fortunately, link #2 above did all the work for me. Since I found it long after installation, I didn't do the "customized distribution" bit, but I DID grab the Domino init script. Once I put that in place, modified the variables to point to my directories and user, and followed the other instructions, I was all set. Now I can control Domino via "service domino (start|stop|restart)" like you'd expect.

Due to the nature of Linux, the question of where to keep the program and data directories is a weird one. The defaults are something like "/opt/ibm/lotus" and "/local/notesdata", respectively, which would certainly WORK, but don't fit in with much else. I've fiddled with my placement a bit and settled on the same thing for the former, but "/var/lib/domino/data" for the latter. That allowed me to set up "/var/lib/domino/daos" and "/var/lib/domino/logdir" (for transaction logging) and keep things relatively clean. One thing to note: if you switch around the directories after the fact, make sure to change both the init script and the notes.ini in your program directory to reflect the new locations.

All told, it seems to be working pretty well. When you're using it like a normal Notes server, there's not much distinction other than the lack of OLE. It has all the same bugs (like the aggravating XPages Java classloader "X is not compatible with X" errors I run into constantly) and same features, which is really the idea.

Pretty URLs

Mar 21, 2011 8:09 PM

Tags: domino

Way back when I wrote my own blog back-end, I went out of my way to make decent-looking URLs that didn't betray the use of PHP for the code. It's not that using PHP was inherently bad, but the ".php" in the URLs was ugly and would have made it tough to move to any other back-ends. It's just one of those good-idea cleanliness things. Back then, I did it with Apache MultiViews and just had names like "archive.php", so I could make URLs like "/archive/2002/12" for that month's posts.

Since I've started using Domino, however, it's gotten a bit tougher. The parts of the URL within a Notes database can actually be pretty great - something like "/People/Foo+Fooson" is about as good as you'd want. However, it's the database path that kills me. You can set up URL substitution rules, but that's a weird combination of setting up a Directory Web Rule plus making sure that every place you make a URL in your code uses that name. For something like a view column, that'd mean either hard-coding your server-specific setting or sucking it up and using @WebDbName. For the most part, I've given up and gone with the latter, for the sake of portability.

XPages bring good news and bad news.

The good news: your application almost never needs to care about the database file path, ".nsf" extensions, or anything of the like. There may be edge cases where you do, but for the most part you can just start your URLs with "/" and the database path is filled in for you (at the cost of starting non-DB paths with "/.ibmxspres/domino").

The bad news: even though the database path reference is now handled by the runtime and not the programmer's code, I don't know of a way to override the behavior. I'm sure it's there SOMEWHERE, buried in the hordes of Java classes and method calls spawned for each page request, but I don't see it. What I'd really want is to tell the XPages runtime that, while "/wow/forums.nsf" is the "right" way to reference the database, I'd rather it just start with something like "/forums". It seems like something that must be handled in the xsp.properties file, but I haven't found anything about it yet. Admittedly, it's tough to search for - most pages with the keywords I want are talking about @WebDbName replacements in SSJS and how to references external Domino resources. If it's not present already, maybe it's something they'll add in a future point release. One can hope.

Tivoli

Feb 13, 2009 12:40 AM

Tags: domino

My company is currently dealing with a client that wants us to look into Single-Sign-On options to integrate with their portal. Since this client has a lot of money, I've been doing just that lately. The keywords are straightforward: they want single-sign-on using SAML. Gotcha!

Step one: search for the keywords. There are a couple promising links:

The first one is an overview article that has two followups, but the only one that mentioned SAML is the first, and only in passing. Right. The second one is a notes.net forum posting asking if it's possible to use SAML to authenticate with a Domino server. No replies.

My further searching led me to Tivoli Federated Identity Manager. I've known of Tivoli primarily as that "other" group of files when I go to download something on IBM's site, part of the sea of giant "IBM Application Services for Application Server Version 6.1, for IBM Application Services 6.1 Enterprise for e-business for Windows XP, Windows 2003 Multilingual (1 of 3)" names.

But the page I found, part of a help DB for the program, contained a kernel of promise: Domino is listed as one of the usable directory stores. Now, I'm not sure if that means this thing will allow you to sign INTO Domino, or if it will just let you sign into OTHER stuff using names FROM Domino, but it's worth investigating, right?

So skip ahead a while and I'm looking at an installer for Tivoli Access Manager. Well, I THINK I'm looking at an installer. I'm also looking at a bunch of other potential installers:

Installers?

So I take a stab at it. "install_ammgr.exe" must stand for "Install Access", uh, "Mmanager", right? Double-click and... nothing. I try some of the others. Double-click and... nothing.

So I put it aside for now. I find some other downloads, such as one that seems to be what I was originally looking for, the Federated Identity Manager. I had a horrifying time with this, including a foray into WebSphere, before I decided to come back and take a whack at Access Manager again. I found the problem this time! You see, since IBM decided that useful file names are for wusses, their downloads are named things like "C1AV9ML", and I, as I always do with Domino, renamed it something useful, "Tivoli Access Manager Base - C1AW2ML", and extracted it into a folder of the same name.

If you're familiar with IBM, or if you have arrived at the present day via a time machine from circa 1990, you may see the problem immediately: the spaces. I mean, sure, Windows has had spaces-in-the-filename support since at least 1995, so not accounting for them was an amateur mistake fourteen years ago, but here we are. I renamed the folder, clearing out spaces, and voila! The installer launched!

It was very promising, too, asking me what directory I wanted, what the Domino server name was, and everything. But when I installed it, it failed, telling me only that it failed and that I should probably check a log file (which I guess it was unable to do itself). When I did, I noticed this suspicious text:

HPDHZ0021E   This file could not be found:  M:\Desktop\IBM\SAML\TAM-Base-C1AV9ML\windows\TivSecUtl\Disk

Oh god. What is in the "TivSecUtl" folder, exactly? Surprise surprise:

Disk Images

Excuse me? There's a folder in the directory tree that was in the zip file that IBM made that it simply can't use because they don't know how to escape spaces? Let me reiterate:

IBM's installer can't work out of the box.

There's already the whole issue that they use these horrible Java-based installers for their "Enterprise" apps. When the main problem is how weird it is that Lotus Notes uses a platform-native installer while Lotus Domino uses something that looks like it was mugged by X11, it's just weird. But when the JVM initialization takes up as much time as the entire rest of the installation, and you have to sit through it half a dozen times because you keep having to make more changes to the installer script, it's like IBM is punching you in the face.

The only theory I can come up with is that IBM doesn't actually want you to use their software. What other explanation is there for an installer package that simply doesn't work? It's not a OS version issue, or a weirdo setup issue, it just doesn't work.

This is even beneath the callous explanation that they do it so they can charge for support contracts. The RIGHT way to lock customers in to painful support is to at least make the program INSTALL and provide a basic level of functionality that is ALMOST what people want, but then make getting from there to the goal like wading through an ocean of steel briars. Like Notes, for example. At least that shows a basic level of respect, not just letting the programmer zip up whatever in-development folder he had and uploading it to the software catalog.

Bah!

It makes all the stupid crap like mysterious line breaks in rich text on the web seem almost user-friendly.