It's much to my chagrin that it took me until this past weekend to properly dive into custom renderers. I had seen them before, primarily when working with mypic, but always avoided building my own, primarily because of what a hassle they are to write. However, I'm now convinced that the hassle is worth it, at least for some primary uses.
If you're not familiar with custom renderers, they're sort of an odd beast, but they're the kind of thing where the concept eventually "clicks" in your mind at some point. In XPages, a component (such as a link, a panel, or an application layout) consists of two main aspects: the structural code that defines the properties of the component and the renderer that handles actually outputting the HTML for the browser. What adding your own custom renderers allows is for you to continue to use the same XSP markup to define the control, yet have very different results.
The best example of this (and the source of most of my education) is the recent Bootstrap4XPages project. It takes a standard OneUI applicationLayout-based app and styles it using Bootstrap markup instead of OneUI, even though the XSP markup is the same. In the normal case, this just means that you can reuse some of the basics when building a Bootstrap-targeted app. And that's good enough on its own: it saves you tons of work.
But the more important aspect is that it lets you focus much more on solving the task at hand and much less on writing to the layout framework.
This is a huge advantage for XPages (and any framework with this separation). As low-key as Bootstrap is (and other modern CSS frameworks are), you still have to structure your markup and sprinkle class names around to match its expectations. With a really fleshed-out set of custom renderers, though, you could potentially not care at all about the resultant UI framework when writing your app and instead focus on using stock or ExtLib controls to solve the problem directly, letting the theme+renderers do the work.
In reality, there'll be a bit of leaking (for example, using Bootstrap-friendly FontAwesome classes instead of images doesn't really fit with standard IBM controls), but I aim to minimize that as much as possible in my future projects. My goal is to use applicationLayout and standard controls for as much as I can, and instead writing a set of custom renderers for each primary theme I want to apply.
I'll have more to write about this later, but for now, the TL;DR version is: custom renderers have the potential to dramatically improve the way XPages apps are written, as long as you're willing to do the initial Java legwork.