Posts for thread "Darwino", by date ascending

An Overview of Darwino for Domino Types

  • Apr 14, 2016

So, Darwino! I've mentioned it quite a few times on Twitter and, particularly, in person, but I think it's high time I write some proper blog posts about it.

To start with, I'll cover what Darwino is. The short version is it's a Java-based development framework with a replicating document database. The interesting aspects go beyond that, though:

  • In addition to Java web servers, it targets mobile devices, both Android and, through RoboVM, iOS. Those devices store their own replicas of the databases for offline work in the same conceptual way as Notes, but with native (or hybrid web, if you're so inclined) mobile user interfaces.
  • The document database sits on top of SQL servers. Many modern SQL servers have native support for JSON data, and Darwino takes advantage of this to get document-DB flexibility with SQL features.
  • Business logic is shared between platforms. Because Java acts as a common language between each platform, and the document DB works the same way locally and remotely, the core business logic of the app can be identical across each targetted platform, with only the UI changing between them.
  • Along those lines, Darwino isn't prescriptive with the UI: it's not a front-end framework itself, instead providing the basis for using other front-end tools, such as Ionic, JSF, and Vaadin for web/hybrid UIs and the native OS toolkits on mobile.
  • The Darwino syncing protocol is designed to be adaptable to other services. This is immediately notable for Domino developers, but can also be (and has been, in some cases) adapted for arbitrary other back ends, like Connections social data or other databases.

How does this relate to Domino/XPages development? That depends on your desires, really.

In some ways, it doesn't. Darwino is its own platform, running on Java web servers like WebSphere and Tomcat, using independent SQL servers like DB2 and PostgreSQL. Darwino's replication between the server and mobile devices is similar to Domino's, but is its own thing. Similarly, the document model, though conceptually similar to Domino (including enhanced reader and author fields), is not NSF.

However, there are a number of reasons why it's of interest to a Domino developer, and the most immediate of those is its ability to do two-way replication with Domino databases. I'm a little biased on this point because of how much time I've spent working on it, but this replication is capable of some nifty tricks to make it capable and adaptable, including transformation of the data, two-way maintenance of document time stamps, and so forth. With this syncing, it makes it very practical to extend your existing Domino app - be it a classic-style Notes/web app or an XPages one - with a Darwino-side UI that uses the same data, synced down to mobile devices for offline access. And this doesn't require migration; since the changes replicate back, the app can remain chugging away unchanged on the Domino side if desired. This also can be tremendously useful for reporting, by syncing the data over to a full SQL database that can be viewed and queried by normal tools.

And, really, it's also of interest to Domino developers personally by virtue of being a platform that has learned a lot of valuable lessons from Domino and extended them in new ways. Those years of accumulated document-database knowledge will carry over nicely, with some extra benefits if you're SQL-familiar too. Any Java knowledge will come in handy immediately, as Darwino is thoroughly Java-based on all target platforms. And, thanks to its pedigree, a lot of the platform support concepts are similar to aspects of XPages (the good parts). In general, the more XPages development you've done, the more it will benefit you (especially if you want to use JSF for the UI!). You could also, with some servlet-implementation limitations, run Darwino apps on Domino via OSGi, and I've been putting some side work into accessing Darwino databases from XPages directly - Darwino could make a solid basis for Domino-run apps.

So this turned into a bit of a sales pitch, but there's no getting around it - I find this thoroughly compelling and exciting. As I have time, I plan to expand on Darwino's various capabilities (along with the other various blog series I still plan to get to). For now, you can register for and download the Community Edition, read the documentation there, and/or track me down with any questions.

Darwino for Domino: Replication and Data Format

  • May 11, 2016

One of the key points of interest in Darwino for Domino developers is its two-way replication. Darwino's replication system was built in such a way that, in addition to its own internal needs, you can also write a replicator to connect to an entirely-unrelated system, as long as that replicator translates the foreign data to and from JSON documents. Domino is a perfect case for this, since the data model is already very similar, and its replicator ships with Darwino and has been a focus of my attention for a while.

Behind the scenes, this replication takes advantage of a lot of the same kinds of things that Domino's replication always has: UNIDs, sequence IDs, original-vs.-in-file modification/creation, and deletion stubs. These details are transparent to the developer: the Darwino adapter knows how to fetch the appropriate data from the NSF and to convince Domino that the Darwino DB is (almost) like a remote Domino server, at least as far as the stored data is concerned.

The primary differences show up in the way data is formatted for storage. Darwino uses JSON for its document format, which has a couple key advantages and disadvantages compared to NSF's "bag of items" approach. The best approach is probably to provide an example and highlight the pertinent differences. Say you have a Domino document that (conceptually) looks like this:

FirstName
	Type: TEXT
	Value: Foo
LastName
	Type: TEXT
	Value: Fooson
Username
	Type: TEXT
	Flags: NAMES READWRITERS
	Value: CN=Joe Schmoe/O=SomeOrg
Birthday
	Type: TIME
	Value: 1970/2/1
Vacations
	Type: TIME_RANGE
	Value: 2016/1/1-2016/1/5, 2016/3/3-2016/3/5
IsAdmin
	Type: TEXT
	Value: Y

On the Darwino side, that would potentially look like this:

{
	"_writers": {
		"username": ["cn=Joe Schmoe,o=SomeOrg"]
	}
	"firstname": "Foo",
	"lastname": "Fooson",
	"birthday": "1970-02-01",
	"vacations": [
		"2016-01-01/2016-01-05",
		"2016-03-03/2016-03-05"
	]
	"admin": true
}

There are certainly a few things to take note of here. First and foremost is the structure of the authors field. Because JSON doesn't have field metadata, the way Darwino does its readers/writers security is by using specially-named and -structured properties within the JSON, and so the converter moves all readers and writers fields into that. This has internal-implementation reasons, but I think it's also conceptually preferable to, say, having a multi-level object within each field to declare its flags separate from the value. The name also happens to be stored in LDAP style, because Darwino is more at home with standard LDAP conventions for that sort of thing.

Another thing to note is the format of the date fields. Since JSON doesn't have a real date/time type of its own, these values are converted according to ISO 8601 and stored as strings. That means that your Darwino application will need to know that those string values represent dates, but they're reasonable to deal with.

The multi-value date field leads to another important aspect: arrays. In Domino, most items are conceptually presented as arrays, regardless of whether they contain single or multiple values, leading to code that requires either explicitly asking for the first element or jumping through hoops to deal with single or multiple values. Since that's a drag to worry about with JSON, the default behavior for single-value items transferred to Darwino is to store them as "bare" values. When configuring the translation (which will be fodder for a future post), you are able to specify that you want a field to be always stored as an array, which will allow the Darwino-side code to be simpler.

The last field shows off an outright advantage of the JSON format: boolean storage. When defining the conversion, you can specify a field as boolean and provide what the true/false values will be, and they will be sent over to JSON as true and false explicitly. That's not a night-and-day change, but it is a nice help.

Finally, there's the matter of rich text, which is unsurprisingly a nontrivial problem. This is handled in an XPage-alike way: MIME rich text is transferred with only minor adjustments, while Composite Data is converted to HTML and cleaned up a bit before transfer. Darwino supports the concept of attachments natively, and so Domino attachments are brought over with a naming prefix to match the field they're attached to, plus a delimiter to indicate whether they're normal attachments or embedded images. The way it is presented on the user end is dependent on the application, but Darwino has some routines to translate storage-safe inline image refs to app-relative URLs.

Later, I will go into the process of how the Domino-Darwino adapters are set up. The short of it is that you create scripts that can run the gamut from just telling the server where to find the NSF to customizing the translation of each field encountered. This allows you to either transfer the data back and forth in the default "best approximation" approach or use the opportunity to enforce a bit of a schema on old data.

Darwino for Domino: Domino-side Configuration

  • May 16, 2016

In my last post, I mentioned that a big part of the job of the Darwino-Domino replicator is converting the data one way or another to better suit the likely programming model Darwino-side or to clean up old data. The way this is done is via a configuration database on the Domino side (an XPages application), which allows you to specify Database Adapters that configure the translation. While it is possible to write these in Java, the primary way is to use a Groovy-based DSL script.

The simplest form of this script may be one line just defining where the NSF is:

nsfName "foo.nsf"

With that, the replicator will open foo.nsf and attempt to replicate all documents with "best fit" translations of each field it comes across. Things can get a little more complex, though:

form("SomeForm") {
	excludeField "foo"
	excludeFields "IgnoreMe"
	
	field "foo_(.*)", nameRegex: true
	arrayField "bar_(.*)", delimiter: "\$", zeroBased: true, initialIndexed: false, prefix: false, compact: true,
		userDataFormatName: "ODAChunk", nameRegex: true
	field ~"impliedfoo_(.*)"
}

form("ArrayTest") {
	arrayField "FirstName", type:TEXT, delimiter: "_"
	arrayField "LastName", type:TEXT, delimiter: "\$", zeroBased: true, initialIndexed: false, prefix: true, compact: true
	
	// Similar to the ODA style
	arrayField "UserData", type:USERDATA, delimiter: "\$", zeroBased: true, initialIndexed: false, prefix: false, compact: true,
		userDataFormatName: "ODAChunk",
		toDarwino: { value ->
			// The value will be an array of byte arrays, which should be strings
			println "testrep: called toDarwino!"
			value.collect { bytes ->
				new String((byte[])bytes, "UTF-8");
			}
		},
		toDomino: { value ->
			// We should provide an array of byte arrays
			println "testrep: called toDomino!"
			println "value is ${value}"
			def result = value.collect { string ->
				string.getBytes("UTF-8")
			}
			println "result is ${result}"
			return result
		}
}

form("RestrictedFields") {
	restrictToDefinedFields true
	field "KnownField"
	field "KnownField2"
}

The specifics of what's going on in this example (pulled from unit test data) aren't too important, but it demonstrates the customizability that an in-language DSL brings. Since the code is executed in Groovy, it has access to the full Java runtime (including, if you deliver it in a plugin, your own custom classes and dependencies), as well as Groovy's nice abilities like closures. In fact, a great many properties, like the converters above, can be specified as closures and executed in the context of each document as it's replicating, allowing for pretty fine-grained translation.

And personally, adapting Groovy into the process was an interesting exercise. Since Groovy was designed explicitly as a "scripting" variant of Java, the process of working it in to an existing Java code base is very smooth, and there aren't too many gotchas. I wrote some Java classes that provide the context for the root and individual "form" blocks, wired up the interpreter, and then they call each other seamlessly. Other languages could probably suit the job well too - JRuby, Rhino, etc. - but Groovy is mature, purpose-built, and largely syntax-compatible with Java itself, making it a very comfortable fit.

Darwino for Domino: Conceptual Overlap and Distinctions

  • Jun 1, 2016

I've talked a bit so far about how Darwino related to Domino from a development perspective, but I think it'd also be useful to delve into specifically which concerns the two platforms address, to see where they overlap and where they don't.

There are two main categories to cover, since Darwino inherits Domino's unusual trait of pouring over from "database" to "app-dev platform".

Database

As I covered a few posts ago, the two are similar at a conceptual level, both being replicating document databases with document-level access control. Aside from the difference between an NSF note's format and a JSON document, the main distinction is that Darwino doesn't cover the actual physical storage of data. Instead, it is based on top of existing SQL servers of various stripes (PostgreSQL and SQLite being the most common). A Darwino application creates a series of tables and uses them as the backing store for the conceptual document database.

This has a number of implications. The main one is that there isn't a "Darwino server" as such - instead, there are SQL databases and Darwino applications acting in tandem. In developing an application, this isn't generally a concern: the Darwino APIs are the same across each database, in the same sort of way that a Domino application doesn't care about the ODS version. However, being backed by a SQL server has some distinct advantages: the server can be administered and optimized using the same knowledge you would use for a "normal" SQL-backed app, and the ability of modern DBs to index JSON data opens up a world of possibilities (think NSFDB2, except good).

The flip side of this bleeds into the second category, as it means that a Darwino application consists of at least two parts: the SQL database and the application, which veers from Domino's "everything in one package" promise slightly.

Application

Things diverge most significantly (though at least as promisingly) when it comes to the application level. Domino has a few "official" ways to develop applications (Notes, legacy web, and XPages) and then hooks to sort of act like a Java EE server, albeit with some notable limitations. Darwino, on the other hand, exists as a sort of "glue layer" in between the database and the application: lower-level than XPages but higher-level than just a database driver.

Darwino provides a common platform for writing Java-based applications, with various services for managed beans, user directories, and so forth, written to work consistently across all of the platforms it targets. Again, this starts out similar to Domino, but diverges in the areas where Darwino takes advantage of other technologies.

At the low level, since Darwino's main requirement is "a Java runtime", it is able to run smoothly on various Java EE servers, on Android, on iOS, and on pretty much anything that provides a capable-enough Java environment (such as, say, Domino). It also, incidentally, means that it works great on Java 8.

At the high level, Darwino doesn't prescribe a specific UI framework, so the field is open to use any of the tremendous array of rapidly-developing Java frameworks on the web side and, as desired, native UI toolkits on mobile. There's a bit of an inherent bias towards REST+client JS applications, since then the same code entirely can be used on both web and mobile (as not every Java web tooklit works on the mobile mini web server as it is now), but that's not obligatory.

Overall

So the overall idea is that Darwino doesn't solve every problem that Domino does, but the problems it chooses to farm out are in the areas where that brings tremendous benefit. In each area where Darwino uses third-party support, it benefits from the tremendous advancements made in recent years, without requiring jumping through weird hoops to get modern techniques to work.