Richard Pajerski Software development and consulting

[February 2023 update: DOTS follow-up, SPR for applet bug]

I recently took advantage of DOTS being back in the Domino 12 server, to replace a Java agent with a scheduled DOTS tasklet and have been pleased with the results.  Using tasklets is generally going to be far more efficient than using Java agents in Domino since a JVM is loaded once with the DOTS server task and remains resident in memory until the DOTS task is stopped whereas with each agent invocation a new instance of the JVM is started.  There are other benefits to using tasklets over Java agents which I may take up in a future post but for the moment, I've run across an issue on a Windows 2019 server installation.

Although the deployment above uses a scheduled tasklet, I was originally hoping to use the triggered NSFDBHOOK events in order to capture some document saves in (more/less) real time.  But while testing on Windows 2019, I noticed that the HOOK_EVENT_NOTE_UPDATE and HOOK_EVENT_NOTE_OPEN events were not being emitted at all or only very infrequently.  I had earlier tested the same tasklet on a Domino installation on a Windows 8.1 client and the events fired more/less as expected.  Aside from the OS difference, everything about the Domino installations was identical -- with one exception: the Domino program installation directory on the Windows client had no spaces but the Windows server was installed in the default C:\Program<space>Files\HCL directory.  Sure enough, after reinstalling Domino on the Windows 2019 server without the space (specifically in C:\Domino), events began firing again.  HCL has also reproduced this and may open an SPR.

In the meantime, after working a bit more with those NSF hook events, my impression is that they are not altogether reliable -- or at least, there doesn't appear to be a one-to-one correlation with each document save/open and a DOTS-generated event.  Some document saves/opens never fire an event.  The source code for the older versions of DOTS is on openntf.org here: https://stash.openntf.org/projects/DOTS/repos/dots/browse/sources but I'm not sure if this is the same code being shipped with the Domino 12 server (though I assume it's pretty close).

If I'm looking in the right place, lines 79 and 80 of the postMessage method (https://stash.openntf.org/projects/DOTS/repos/dots/browse/sources/dotsNSFHook.cpp), have:

STATUS error = MQOpen(queueName, 0, &dotsmq);
 if ( error == NOERROR ){

where DOTS presumably intercepts the necessary events from an internal Domino queue.  But what if there *are* errors here?  Will our DOTS tasklets ever know about them?  Maybe errors are unlikely here but perhaps this is the source of some missed events.

[February 2023 update: DOTS follow-up, SPR for applet bug]

A notable aspect of this bug is that there are no error logs (that I could find) to indicate something's wrong.  Nothing in the Java Debug console, the IBM_TECHNICAL_SUPPORT folder, log.nsf, etc.  You will see the Java coffee cup but I think most end users will not immediately interpret it as a problem:

This issue is probably not an inconvenience for most deployments since Java applets were never widely implemented in Notes (in my experience).  But where applets are used, they can bring quite a bit of power (e.g. custom interfaces for any third-party system with a Java API -- databases, message queues and so on).  The good news is that applets do still work in 32-bit Notes 12.0.2 and will likely be supported in Notes as long as Java 8 remains the default runtime.

It looks like the expiration date for APNS p12 certificates that shipped with Traveler 12.0.1 was on or about December 19, 2022.  If you have a Traveler 12.0.1 (November 2021) installation and started seeing Domino console messages similar to:

on or after that date, the resolution is to apply Traveler 12.0.1 Fix Pack 2 which includes new APNS p12 certificates that expire in September 2023.  Until that fixpack gets applied (or you upgrade to Traveler 12.0.2 which also contains updated certificates), iOS users will no longer receive notifications of new email on their devices.

More here:  What's new in Traveler 12.0.1 Fix Pack 2?

Among the technical reasons Domino has enjoyed longevity in the application server market are its stability and reliability.  But of course technology changes rapidly and being able to successfully adapt to new trends has also kept Domino relevant over the years.  Striking a balance between trendy and relevant is not always obvious but the stewards of Domino have generally taken a conservative approach to introducing and supporting new technology which continues to benefit the product in the immediate and long term.

Very briefly then, beyond the tried-and-true world of Notes client server application development, let's look at three modern technology options developers can choose from in 2021:

1.  Domino AppDev Pack (https://doc.cwpcollaboration.com/appdevpack/docs/en/homepage.html)

From the site:  "The AppDev Pack primarily adds Node.js support to HCL Domino Server."

So developers already familiar with or wanting to explore the JavaScript-centric Node.js have a custom-designed set of tools for programmatically and securely accessing Domino applications and data.

2.  Domino REST API (Project Keep) (https://opensource.hcltechsw.com/domino-keep-docs/)

From the site: "Domino REST API is designed to re-establish Domino as a world class, modern, standards-compliant, cloud native and enterprise-level collaboration platform. It adds contemporary REST APIs to Notes and Domino, enabling a modern programming experience with the tools of your choice."

This is still in beta but will hopefully be out in Domino 12.0.1 later this fall.  Domino has for many years had the ability to offer data via a REST API (Domino Access Services) but Keep modernizes that effort by implementing (among other things) OpenAPI and JWT Authorization.  In addition, developers will be able to introduce their own server-side modules called "verticles" as Keep runs on top of Vert.x.

3.  Tasklets with DOTS - Domino OSGi Tasklet Service (https://help.hcltechsw.com/domino/12.0.0/admin/wn_dotsredux.html)

From the documentation: "DOTS is a generic Domino add-in task that lets users create Domino server tasks by creating a tasklet container using Java OSGi plugins."

Ok, this is not exactly modern -- DOTS was an OpenNTF project that's been available since Domino 8.5.3.  Although officially dropped in the Domino 10/11 era it was recently updated and re-introduced in Domino 12.

Strictly-speaking, DOTS is the Domino server task that manages the tasklets which are Java server-side plugins.  Tasklets use a standard mechanism for starting/stopping (OSGi-defined) which makes them similar to Java agents but have the additional benefits of (a) not being tied to a specific .nsf file, (b) not requiring a full JVM reload on every execution and (c) better access to the Domino C API.

How are you developing Domino server-side code these days?

Having recently installed a Notes/Designer/Admin 12 client on Windows 8.1 Pro on an aging Intel i7 Quad-Core (with eight logical processors), I was surprised to see that after startup, Notes was consistently grabbing one logical processor and pegging it at around 15%.  Windows Task Manager showed that nlnotes.exe was the process and it wouldn't let go of that processor until Notes was closed:

Notes, Designer and Admin clients all worked fine and interaction with local- and server-based applications was normal.  Without giving it too much thought, I tried a few things like removing cache.ndk, stopping a couple Notes-related services and making sure the preference "Enable scheduled local agents" was disabled.

But none of that worked and strangely, Disk and Network utilization were both at 0%:

The client was set up in the normal way by connecting to a Domino server with an existing mail file on the server.  However, this workstation needed to use a Location that sends and receives mail from a POP3 server instead of Domino.  For that purpose I used an existing names.nsf that already had Account and Locations documents in place.  All of that connectivity worked and the mail flowed normally.

The next step was internet search but nothing obvious jumped out and most references to similar problems ended with Notes client crashes which I was not experiencing.

The Location document for this POP3 configuration kicks off replication and runs it every 10 minutes.  I noticed that after starting the client, there was a gap of a few seconds where the nlnotes.exe process was at 0% and didn't go up to 15% until replication started (and then stayed there).  Thinking the culprit was the Replicator, I disabled replication and restarted Notes... but nlnotes.exe was back to 15%!

But this time I could clearly see that process spiked when "Notes configuration settings have been refreshed" scrolled across the Status bar:


Based on that message and the fact that a POP3 configuration like this is not commonly used, I kept pursuing the Location document as the source of the problem.  And the problem was indeed there.

When you configure the Notes client for POP3 mail retrieval, only the "Mail" tab of the Location document needs to be filled out:

The "Servers" tab can remain empty -- and that was the problem!  At a minimum, the "Home/mail server" field MUST have some value in it to calm down the processor:

It doesn't matter if the server is down or the value entered isn't even a Domino server -- nlnotes.exe will report that the server is not responding but it leaves the processor alone after that:

Ok, problem solved, back to work. 

Ray Ozzie is among the 2021 Computer History Awards honorees recognized "For a lifetime of work in collaborative software and software entrepreneurship".

https://computerhistory.org/press-releases/chm-honors-tech-legends-for-lifetime-of-contributions-and-impact-on-humanity/

A virtual event took place to honor Ray on March 18, 2021 and will be generally available in the coming days.  Thank you, Mr. Ozzie for your contribution to computer history!

In an effort to cover some specific cases, a minor feature was just added to CertMatica (3.6.0) to automate copying LE certificates to additional locations on the server's file system.

The idea is to allow Domino to share the LE certificate with one or more services running on the same machine (such as a reverse proxy like Nginx or an alternate SMTP service) without further manual intervention.

CertMatica 3.6.0 Trial

First of all, a big "Thank you!" to all CertMatica customers!

Just out this week, CertMatica 3.5.1 is a maintenance release with simplified switching between Let's Encrypt test and production modes and minor improvements to logging and documentation.  However, for those running on earlier versions of Domino 9.x and 10.x servers, this release also includes an important update to the CertMatica Cacerts Utility which can be used to address potential connectivity errors caused by missing or expired intermediate certificates in the Domino JVM truststore (cacerts).  For information on Let's Encrypt infrastructure changes related to this update see https://letsencrypt.org/2019/04/15/transitioning-to-isrg-root.html.

As always, feedback and commentary is always greatly appreciated!

Since many development and administrative tasks in Notes/Domino can conveniently be carried out with great front-end tools like Domino Designer or Administrator, it can sometimes be inconvenient when we're required to use the command line or terminal to get things done.  Working with Domino keyrings is a case in point and one of the reasons why I developed Aperture.

Aperture is a lightweight desktop application that allows you to work with those .kyr files without having to resort to the command line.  It works with both the KYRTool and OpenSSL to allow you to visually create keyrings, view their contents, create Certificate Signing Requests and several other tasks you'd normally being doing on the command line.

Please visit the Aperture product page for more details:  https://www.rhpconsult.com/aperture.html.

As always, comments and suggestions are appreciated!

If you were a Notes/Domino 11 beta participant, the 11.0.1 preview is now available.  Use your existing links to access the downloads from FlexNet and the beta forum.

is right around the corner (August 7, 2019):  https://register.gotowebinar.com/register/5645329603541153035. 

Be they public, private or the newly-minted "Partner-led" clouds, one message HCL is making clear is that the future for the newly-acquired IBM collaboration product portfolio will be all about the cloud.  Richard Jefts (General Manager, HCL Digital Solutions) has just published here on the new approach: Update on HCL Acquisition of IBM Collaboration Portfolio (Edit March 2021: Update on HCL Acquisition of IBM Collaboration Portfolio).

It appears that the biggest impact will be on current SmartCloud Notes users who will now need to transition away from that offering to one of the new HCL cloud models.  In practice, that will probably entail some form of migration back to a traditional, on-premises Domino/Portal solution (or Domino/Portal-hosted partner solution).  Those who currently run on-premises applications and solutions, including Verse, will (likely) not be immediately affected.

It's good to hear mail will continue to be a core area for Domino; however, the "for the foreseeable future" is an interesting qualification that suggests it may not be for long.  The tight integration of email within collaborative Notes/Domino applications won't go away but will loosen up as different messaging providers take over the role of Notes mail clients.  HCL's announcement here is not surprising (or at least not shocking) and I think it strikes the right balance between managing what works well today on-premises and where much new development will be going forward: cloud.

Having developed with Java for a number of years in various environments (Notes/Domino, Tomcat, ActiveMQ,  Android, desktop, etc.), I was initially skeptical when I read this article and watched the video about the recently-improved Eclipse OpenJ9: https://developer.ibm.com/videos/introduction-to-eclipse-openj9-and-adoptopenjdknet/

Yes, Java has incrementally improved over time but the claims here seemed a bit over the top.  To think I might get both noticeably faster startup *and* up to 50% memory reduction just by switching to J9 seemed to be a bit too optimistic.  But after downloading (adoptopenjdk.net) and giving it a spin, I was not disappointed.

Sure enough, out-of-the-box startup time for Netbeans 8.2 on Windows 8.1 increased dramatically against Oracle Java 1.8.0_191 (running quad-core I7 on SSD).  There was no point in taking measurements -- it was up and ready in three seconds!  This didn't seem possible with Netbeans but there it was.  Everything worked the same as before ... only faster.  Then the real shocker: RAM usage went from roughly 650M down to 268!  Huh?  If I can eliminate that much RAM usage for hosted server side deployments, it's going to translate into real cash savings.

On top of the performance upgrade and memory savings, I immediately noticed that Swing is visually better in J9 than OpenJDK [edit: with the HotSpot VM].  In particular, the default font rendering is really nice!  In the past, OpenJDK has generally lagged behind Oracle Java for desktop applications and still does; but to my eyes, J9 is now at visual parity with Oracle (or perhaps better).

I realize that the J9 has been the JVM in Notes/Domino all these years but I've never attempted to benchmark it against other JVMs since IBM never really promoted it as a JDK for Windows.  I'm currently using 9.0.1 FP10 which uses build 8.0.5.21 of J9:

Hopefully, IBM can manage to get the latest J9 into an upcoming fixpack.  I sure have lots of Notes and Domino Java code that could benefit from it.

A big congratulations and thank you to Mark Stoodley and all the other engineers and players behind this release!

Among the areas getting attention in the upcoming release of Domino 10 is the data store.  As part of making Domino more bullet-proof, removing the 64GB limit on the NSF size is planned and will be a fantastic improvement.  Naturally, this is going to put more emphasis on overall database scalability.  As Domino gets more scalable, I think it's going to need a feature that many of us have been seeking for some time: multi-document ACID transactions.

Being able to save two or more documents as a single transaction cannot currently be done natively in Domino.  The best we can get is saving all of the fields associated with a single document using NotesDocument.Save: the operation either succeeds or fails.  This is fine for most Notes/Domino applications but there are a number of cases where having a transactional save across multiple documents is desirable.  A simple work order system where multiple, related tasks are attached to a main work order request is an obvious example.  Implementing each task as a separate document is an intuitive approach and can simplify programming of such a system.

MongoDB 4.0 recently introduced multi-document transactions.  And although Domino doesn't directly compete with MongoDB or other NoSQL databases, as it begins to scale, it's nonetheless going to need to act more like them.

What do you think?

IBM has announced a multi-year investment in Notes Domino with a major new release (Notes 10) coming out in 2018. The investment will include closely-related products such as Notes Traveler, IBM Sametime and IBM Verse.

Specific product details are scant at the moment but it's encouraging to see IBM laying out a long-term roadmap for Notes and Domino and broadcasting a commitment to protecting its clients' investments. Also, the new direction allows IBM partners to hope for commercial stability for these products for the foreseeable future. Overall, I'm optimistic about this announcement.

However, the partnership with HCL Technologies for future development raises some questions. Will HCL Technologies be able to innovate the way that Iris Associates once did? Is this merely a cost-cutting measure or does IBM no longer have the internal talent to take Notes/Domino into the future (or both)?

More here:
https://www.ibm.com/blogs/social-business/2017/10/25/ibm-announces-investment-notes-domino-version-10-beyond/