The $ViewFormat Structure For Table Views (As I Understand It)
Jul 14, 2021, 4:00 PM
The last week or so, I've been working on reading Notes design elements without my longtime unreliable friend DXL. This is a task I've done in parts a few times before, but now I've set out to finish the job.
For the most part, the task is arcane but explicable: the structures are generally documented well enough in the C API docs, and most non-string items are usually only a few elements long. Views - likely because they're as old as Notes itself - are not one of these easy parts.
$ViewFormat item in a view design note contains, well, the format of the view: the overall settings (like the background colors) as well as the formats for the individual columns. Because views have gotten more complicated over the years, this item has grown. And because these types of structures have to grow in a way that's generally compatible with older code that might read it, the format is essentially an array of progressively-newer structures that alternatingly describe new traits of the view and the columns.
Some of this is described in the C API documentation. In there, you'll find "readview.c", which shows a basic example of how to read a view. Unfortunately, this example stops at the
VIEW_TABLE_FORMAT2 structure, which, as we'll see shortly, is not far in. From what I can gather by looking at the "FirstVer" values in the C API documentation NSF, this file demonstrates only what existed before Notes R4.
So that example is a bust, but fortunately "viewfmt.h" has some more information. Near the top of that file is an overview of the $ViewFormat item structure in general, and it's more helpful. It's not fully helpful, though: it stops with things added in R6, and doesn't cover parts added in 6.5 and beyond. Additionally, it doesn't explain the storage of the R6-era variable data - the hide-when formulas and twistie image references - and those are a bit weird.
The mitigating factor for the documentation is that, while it doesn't explain the order of the remaining data, it does at least include definitions for the post-R6 structures, such as
VIEW_COLUMN_FORMAT5 and whatnot. That is... it mostly includes that: they're fully present in "viewfmt.h", but some are missing in the NSF doc DB, and the NSF lacks some important flags. Still, it's something.
My final task remained: I had a few hundred bytes left and, armed just with some available structures and Karsten Lehmann's earlier attempt, I had to figure out how views work. And I think I did it! I haven't tested all edge cases, and in particular this will be different for calendar-type views, but this holds together. I'll update this as I find more hiccups.
I've been accessing this data as read by using the pointer returned by
OSLockObject on an item value's
BLOCKID. I'm also running on an amd64 Mac. Those two aspects may affect the need for
ODSReadMemory - things may be different if you're on a different architecture, or perhaps the way I read it already brought it into host format. In my use, I found that
ODSReadMemory was harmful in all cases, and that it was best to read the memory directly. Additionally, this is a view created using Designer V12 and read using the V12 libnotes.dylib, so anything newer won't be reflected here.
The rules around
ODSReadMemory elude me at the moment. The way I'm reading this data involves the value
BLOCKID as fetched by
NSFItemQueryEx, which indicates that the value will be in canonical format for types including this. In theory, that should mean that
ODSReadMemory is required for these structures, but it ends up harmful.
To start out with, we have the
VIEW_FORMAT_HEADER structure, which defines the format version (always
1) and whether it's a normal "table"-type view or a calendar view.
This header structure is, in table views, actually part of the
VIEW_TABLE_FORMAT structure that is the true beginning (and is also where things will diverge for calendar views). This one's pretty straightforward, with no variable data, and contains the critical tidbit of the
Columns value for the total number of columns, a number that we will reference again and again. Once read, advance your pointer
Following this will be one
VIEW_COLUMN_FORMAT structure for each column based on the count in the format. For each, read the structure and advance your pointer
sizeof(_VIEW_COLUMN_FORMAT) bytes for each. The first WORD of each of these will be equal to
Next up is the variable data for those columns: the item name (string), the title (string), the formula (compiled formula data), and the constant value data (documented as reserved). These should be read by iterating over the above-read
VIEW_COLUMN_FORMAT structures and reading byte arrays with length based on
VIEW_TABLE_FORMAT2, which is the last bit read by the "readview.c" example. Read
sizeof(VIEW_TABLE_FORMAT2) bytes and increment your pointer.
(At this point in time, you might be inclined to ask "hey, how come
VIEW_COLUMN_FORMAT has a handy signature to identify it but
VIEW_TABLE_FORMAT2 doesn't?". Well, to that I say that you sure are asking a lot of nosy questions, pal, and maybe you should mind your own business. Anyway, this will be a reocurring feature.)
This is followed by one
VIEW_COLUMN_FORMAT2 structure for every column in the view. Though this includes variable-data sizes, we don't read them yet - just read
sizeof(VIEW_COLUMN_FORMAT2) * n in total here. These will each have
VIEW_COLUMN_FORMAT_SIGNATURE2 in the first WORD.
VIEW_TABLE_FORMAT3, which is a blessedly fixed-size structure, so you can just read
sizeof(VIEW_TABLE_FORMAT3) bytes and move on.
Now here's where we come to the variable data from
VIEW_COLUMN_FORMAT2, and where things start getting weird. To read this data, iterate over each
FORMAT2 structure for your columns and check the
wTwistieResourceSize properties. If the hide-when size is above zero, read the hide-when formula first as a compiled formula. Then, read the twistie resource as
CDRESOURCE composite-data structure if its size is non-zero. Fortunately, the lengths for both of these are defined in
VIEW_COLUMN_FORMAT2, so the total amount to read will be
wTwistieResourceSize from that structure.
Next, we have
VIEW_TABLE_FORMAT4, which is a simple little structure. Of note here is that the
RepeatType property - which I gather is how to repeat the background image of the view - references "viewprop.h", but that is not in the documentation. Oh well. Pleasantly, though, this structure includes a
Length property as its first WORD, so read that many bytes (even though it should always be 8).
After this, our friend
CDRESOURCE shows up again, this time as a single entity to represent the background image of the view. Since there's nothing that tells you the length of this before hand, you should glean the length from the CD record header, which is a
WSIG - so the second WORD. Read that, then backtrack and read that many bytes.
Now, we have 0-to-n
VIEW_COLUMN_FORMAT3 structures - one for each column that includes date formatting (from that "Style" dropdown on the fourth tab in Designer). To read these, look over your array of
VIEW_COLUMN_FORMAT2 structures and, for each that has
ExtDate in its
Flags3 field, read one of these. Each column's structure is immediately followed by variable data for the different formatting strings, so read strings based on
DTTsepLen from the structure, and then move on to the next applicable column.
These are followed by 0-to-n
VIEW_COLUMN_FORMAT4 structures. These are the same idea as above, but for number formatting. Look for
VIEW_COLUMN_FORMAT2 entries with
NumberFormat in their
Flags3 and read a
VIEW_COLUMN_FORMAT4. Each of these is followed immediately by strings based on
CurrencySymLength, so read those too before moving to the next applicable column.
We've got one more of these in this sequence, and that's
VIEW_COLUMN_FORMAT5: formatting for names columns. Iterate through your
VIEW_COLUMN_FORMAT2s and look for
Flags3 and read these structures. Each one is followed immediately by the programmatic name of the column containing the distinguished person name for Sametime purposes, as defined by
wDistNameColLen. Fortunately, this structure also includes a
dwLength (which is a WORD despite the prefix) that contains the total length of the structure and its associated variable data, so you can read that many bytes in total here. It also, though, contains a nasty trick: you might be inclined to think that this structure also contains the name of the shared column used here, what on account of
wSharedColumnAliasLen being in the structure and the documentation saying that's what it means. You'd be wrong, though - that value will be 0 and you'll get nothing from it.
This next part is a doozy. The fact that
VIEW_COLUMN_FORMAT5 purports to contain the name of the shared column but in fact doesn't raises the question of where that name actually is. Well, it's right up next - you just need to read the string! But hrm: the length value in the previous structure was 0 and, besides, there won't even be one of those for non-names columns. Is it null-terminated? No: your next bytes will likely be something like
0x07 is not a column name, that's for sure. What we have here is a P-string, prefixed by a WORD containing the number of characters. These strings represent both shared column titles and any titles for columns marked as "Do not display title in column header", with the latter first. So, loop through your
VIEW_COLUMN_FORMAT2s again, looking for
Flags3, and, for each match, read a WORD length and then that many characters to glean the title. Then, do the same for columns with
Flags3 for your shared-column aliases.
The final structure in play that I know of is
VIEW_COLUMN_FORMAT6, which fortunately is handled in basically the same way as formats 3-5. Look through
VIEW_COLUMN_FORMAT2 again, this time checking
ExtendedViewColFmt6 (I appreciate the functional name), and read a
VIEW_COLUMN_FORMAT6 for each. This one again has some variable data immediately after each structure and also has a
Length property, so read in that property's value in total bytes. This structure isn't in the doc NSF, but it is in "viewfmt.h", and it contains R8-era composite-app stuff.
Finally, it seems like there's a trailing 0. It makes me nervous that I end up with a single byte, but it seems to be consistent, and so my guess is that it's here to meet a WORD boundary, which is extremely common in the Notes API.
Incidentally, I don't know where
VIEW_TABLE_FORMAT5 comes in, if at all. It's in the NSF and "viewfmt.h", but looks like just a duplicate of
VIEW_TABLE_FORMAT4. It's possible that it sometimes shows up right after 4 when some specific thing is set, though it has no signature component and would be difficult to detect if so.
The Structure (Condensed)
If you came here to just get a quick overview of the format, I'm afraid I've pulled a "recipe blog post" on you and wrote it all in prose before the important part. In any event, here's the above but condensed:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
VIEW_TABLE_FORMAT VIEW_COLUMN_FORMAT * column count (variable data for each column) // Order: (item name, title, compiled formula, constant value), repeat per column VIEW_TABLE_FORMAT2 VIEW_COLUMN_FORMAT2 * column count VIEW_TABLE_FORMAT3 (hide-whens and twistie resources) // Order: (compiled formula, CDRESOURCE), repeat per column where VIEW_COLUMN_FORMAT2 has wHideWhenFormulaSize > 0 or wTwistieResourceSize > 0 VIEW_TABLE_FORMAT4 CDRESOURCE // Represents the background image for the view VIEW_COLUMN_FORMAT3 * x1 // One per column where Flags3 has ExtDate. Order: (VIEW_COLUMN_FORMAT3, var data), repeat per column VIEW_COLUMN_FORMAT4 * x2 // One per column where Flags3 has NumberFormat. Order: (VIEW_COLUMN_FORMAT4, var data), repeat per column VIEW_COLUMN_FORMAT5 * x3 // One per column where Flags3 has NamesFormat. Order: (VIEW_COLUMN_FORMAT5, var data), repeat per column (hidden column titles) // One per column where Flags3 has HideColumnTitle. Stored as WORD-prefixed P-strings (shared column names) // One per column where Flags3 has IsSharedColumn. Stored as WORD-prefixed P-strings VIEW_COLUMN_FORMAT6 * x4 // One per column where Flags3 has ExtendedViewColFmt6. Order (VIEW_COLUMN_FORMAT6, var data), repeat per column
Hoo boy, this was a fun one. Anyway, assuming this holds up under further testing, my hope is that this post will be useful for the next person to come around to try to decode $ViewFormat. If that's you: good luck!
Ben Langhinrichs - Jul 15, 2021, 2:40 PM
I've worked with these structures a lot, but I'm not sure I've ever tried to document them better. I'll see if my random notes in the code match up with the stuff you've found, but it seems as if you've gotten past the worst of the gotchas. Midas and Exciton both let you use these structures to render to HTML, but I only allow a little flexibility in generating views programmatically because there are so many undocumented combinations of things when you get to hidden columns and formula-hidden columns and so forth. Thank you for taking the time to try to put this down in an organized way. I hope HCL takes a hint and puts more of this in the V12 C API. It is awful trying to guess and deduce it all.