Women are now allowed to choose their own husbands – I disagree with that idea

My friend and administrator-extraordinaire Chris Miller objects to IBM’s decision to publish the Domino Eclipse Update Site template under the Apache License on OpenNTF.org. We’ll get to the specifics about his objection in a moment, but I wanted to start with the very obvious matter of objecting to increased freedom. Making commercial code available under an open source license is increasing the freedom of customers, the freedom of partners, and the freedom of the public to make their own choices. Philosophically, disagreeing with this idea is like disagreeing with marriage equality or competing cell phone carriers or being able to turn off your television. Releasing code under an open source license does not force anyone to customize its use; it simply allows people to choose to modify and reuse it.

Chris continues…

I think any system type database should remain under IBM direct controls and not be open. I also know that this database has not really been updated with enhancements since creation which leads to another whole posting on support for the plug-in model

There are a few misconceptions revealed here. The first is that open sourcing anything takes it out of the author’s direct control. That is simply not true. The Eclipse Update Site template belongs to IBM. They hold the copyright, and they have direct and final control over any changes to it in their shipping code from now until the heat death of the universe.

Making it open source means that now other people have permission to copy and change it. That’s all it means: “If you take all or part of this code and give it away or sell it, we promise not to sue you.” That is the end of the implications of open sourcing it for IBM.

Incidentally, the template HAS been updated with enhancements since its creation, as the original version did not have the “disable feature” and “enable feature” that the 9.0 version does.

Chris goes on to ask the question…

Should this or all system databases go open source to allow you to tweak IBM Domino server installs?

Again, this is a misunderstanding of the decision to open source. You have ALWAYS been allowed to tweak Domino server installs. Or at least as far back as version 3, when I first worked with customers to build customized install kits. All the system templates controlling the behavior of Domino servers and domains have shipped with all source code available since the inception of the Notes platform, and the power to customize these templates has been a hallmark of the platform for over 20 years.

What releasing a template under an Apache License does is allow people to redistribute their customized templates to other parties without specific permission from IBM. That’s all that’s changed. And because of this change, Karsten Lehmann has already released an enhanced version with new features that empower partners and customers, should they choose to use it.

When I pointed this out to Chris in the comments, he replied…

That was the whole point actually. Wonderful improvements have been made and numerous awesome templates are there. The point is IBM should update frequently and control system databases. It very well might take ten years, but it shouldn’t… We shouldn’t RELY on outside volunteers to improve defining system files on a paid product.

I forgive Chris for not having the intimate knowledge of the origins of OpenNTF.org that I do, since he does not have direct access to my brain, but I can tell you that this idea of frequent updates to system databases was the EXACT REASON why Bruce and I founded OpenNTF. It is the EXACT REASON why Vince put so much effort into the flagship project on OpenNTF: the OpenNTF Mail template, an enhanced version of the standard Notes 6 mail template. It was this project that created our first breakthrough success and delivered enhancements that people had been requesting for almost a decade.

And it was totally illegal.

We were lucky that we had some notoriety as supporters of the Notes development community. We were lucky that we had the support of people like Ed Brill to help us communicate with IBM. We were lucky that all this happened before the SCO lawsuit got really expensive and turned IBM legal process into the infinite slow death by sarlacc that we know today. Because IBM could have sued us for everything we had at any time. (This fact is exactly why OpenNTF didn’t have any legal entity for about 5 years.)

What these open source releases mean is that IBM is disempowering themselves from suing partners and customers if they do what we did.

As I said, the point of creating OpenNTF was to demonstrate to IBM that they could build, test and deliver template updates outside the existing version release process. We were successful in this strategy with exactly ONE template: the Domino Blog template. By that measure, we completely failed. IBM didn’t change their release process for Domino software until the XPages team built the XPages Extension Library as open source first in 2010, and then later delivered it as a supported Update Pack to Domino.

It took 8 years to get IBM to change their approach to delivering accelerated customer value on the Notes/Domino platform, and even now there are only a handful of projects that live as IBM-contributed open source that then roll back into the shipping product. Most of the key templates remain officially in the legal limbo of “we own it, but you can access all the source code with our free version of IBM Domino Designer but you can only use any changes you make internal to your organization unless we give you special permission or turn a blind eye.” Not exactly a great foundation for a vibrant commercial ecosystem.

Of course, as we know from those projects that IBM releases on OpenNTF, open sourcing parts of Domino doesn’t RELY on volunteers for improvements at all. We can see this is true by looking at two simple facts: 1) only IBMers release new versions of IBM-contributed projects; and 2) almost never do releases include any code whatsoever from a non-IBMer.

Chris continues…

Plus this adds complexity when calling support. Just say the words custom template (OpenNTF or not) and the response is the same. Replace that with the supplied default, test it and let us know.

I’m not sure how this is an argument again IBM giving customers and partners permission to modify and distribute IBM’s code. IBM Support doesn’t care whether your custom template comes from OpenNTF, a business partner, an ISV, ISSL, or your own in-house developers. All they care about is whether it’s exactly what’s downloadable from Passport Advantage. So this is just an argument against changing anything from IBM ever. And I get the theory behind the argument — because so much as adding a “Groups by Members” view to the Domino Directory effectively “voids the warranty.” We’ll come back to that warranty in a minute.

My other friend and uber-admin Gab Davis also replies in the comments…

I’m 100% with Chris on this. As an Administrator who is in overall charge of security of the servers I manage I very rarely get to deploy anything from OpenNTF as customers consider it “outside” IBM. If I take a template from IBM I know there isn’t anything nefarious in it (or at least not deliberately nefarious) , I don’t know that with anything that comes from outside IBM…

Again, this statement reflects some fundamental misconceptions about what actually happens with Domino software. Firstly, with IBM templates, the only way you know there isn’t anything nefarious in them is if you read the source code they provide. IBM is certainly capable of writing and shipping software with bugs, just like every other software publisher on the planet. And some times these bugs can be quite nefarious, resulting in user confusion, data corruption and even security compromises. To the extent that you have the source code that is executing at the time the bug is revealed, you can discover for yourself why it’s happening. Otherwise, you don’t know anything more than IBM staked their corporate reputation for quality on the results. (The reader is free to decide for his or her self what this reputation is worth.)

Secondly, anything coming from OpenNTF (or any other open source group) comes with the source code. So you can, in fact, know whether anything nefarious is going on, because you and everyone else on the planet can read the instructions being compiled into the byte code that instructs the machine. If you’re in doubt about whether the object code matches the source code, you’re free to recompile the project yourself after a full audit.

Gab continues…

…nor do I have the reassurance that whoever wrote the enhancement understands all its ramifications on the server. How can they when Domino itself is not open source.

As someone who’s personally contributed more code to OpenNTF than almost any other single person, it’s difficult not to take this statement as an attack on my professional competence. Gab is a world-class expert on the IBM Domino server with decades of experience, so she knows a lot about the ramifications of running code on it. I am also a world-class expert on the IBM Domino server with decades of experience. I know a lot about the ramifications of running code on it, too. After 22 years, I know a lot about what’s happening when Domino servers operate, and I bring the full breadth of this experience to bear in every line of code I write for OpenNTF projects. I also have the advantage of being able to read documentation, collaborate with dozens of other world-class Domino developers in real time, ask questions of IBM engineers directly and when absolutely necessary, read the source code for parts of Domino itself.

Plus, because it’s open source, on those rare occasions (okay, not-rare-at-all) when I DO write a bug, there are dozens of people testing my code, discovering the exact thing I’m doing wrong, and then either fixing it themselves or explaining to me exactly what my boneheaded mistake was (usually with much rejoicing.)

And this happens for everything on OpenNTF. That’s why being able to report bugs and post code back to a project was one of the very first features we created on the website back in 2002. And it’s why the most prolific contributors to OpenNTF are people with vast experience, many of whom, like Gab, are IBM Champions.

Gab goes on…

Experience has also taught me that if your server has a problem and there is anything considered “3rd party” on it, IBM support will instruct you to remove / disable that before proceeding.

Experience has taught me the same thing. In fact, there’s an even wider principle that experience has taught me: that any software vendor will blame customer problems on other software vendors. The fact that the 3rd party code comes from OpenNTF is no different than if it comes from TeamStudio or GSX or Blackberry or Apple or Microsoft or Google or Red Hat; if you’re doing anything other than running exactly what a single vendor delivers on that single vendor’s platform, they will avoid accepting blame for problems that occur in your environment.

I’m going to assume that Gab has customers who use Linux. Or Domino Designer. Or the Notes client. Or XPages. Or Android devices. All of these are either whole or in part based on open source software. She probably also has customers who use Microsoft Windows, whose engineers not only don’t have the Domino source code, but have a competitive incentive to introduce nefarious code into their platform to disrupt Domino’s operation. (Remember “Windows ain’t done ’til Lotus won’t run?”)

The tendency of vendors to blame 3rd party code for problems is a reason to only run the Notes basic client on OS/2 clients against Domino on AIX or i servers. Any other configuration means that Gab’s customers have already accepted the possibility that IBM will blame other parties in the support chain, so using code from OpenNTF hasn’t changed their practices at all.

Chris returns to the conversations in a further comment…

It isn’t them saying we don’t support it anymore. It is the fact of using these variations will not be supported when you call. Why not spend the energy and resources to expand these as part of the paid product?

This takes us to the heart of the matter: what will IBM ‘support’ when a customer calls? I’m always confused when people say this is an important consideration. They seem to have this perception of ‘support’ where if they run into a problem, they can call IBM, who will be standing by the phone, cape waving in the wind, and instantly spring into action to fly at supersonic speeds directly to the server room of the customer, where they will use their Correcto-vision to fire the proper machine instructions directly into the CPU and thus achieve user nirvana forever and ever.

But that’s not at all what happens, is it? What actually happens is that they call a support intern on the phone and argue for 15 minutes about whether they tried turning the computer off and then on again and tried re-installing Windows. After a while, they get moved to a level 2 person that knows something about the platform, who then asks about 3rd party software and supported configurations. If they pass that litmus test, then they have a shot at having that person say they’ll have a Level 3 engineer or maybe even a Senior Technical Engineer get back to them, and be assigned a PMR. If the next level of support can validate the problem, it then goes on to product management. If other people have reported the same problem, then IBM might decide to assign it an SPR. If it gets an SPR, then a specific product manager will define the scope of the problem, how it impacts new or maintenance revenue, and what engineering’s best guess at the fix cost will be. A senior level engineer or VP will then decide to fix the bug in a specific release. Someone, often an intern or an offshored lab resource, will write the fix and check it in. If it compiles, it will go into a test build. If the test case passes, it will go either to code review or be shipped in an internal release. If it survives enough internal testing, it shows up in a pre-release build. If it doesn’t cause problems in the pre-release, it will make it through to GA code. If it does draw problems, it will go through a senior level code review and either get yanked out of the release or fixed by a more senior engineer. Finally, after years of evaluation, estimates, design, coding, testing, review, beta testing, and further review, it might be made available to the customer.

The customer, of course, isn’t going to deploy the code on the first day. In fact, if the new code ships in a .0 release, they might never deploy that release at all, choosing instead to wait for the first round of bug fixes. If they choose to deploy the first maintenance release, they’ll then deploy it into an internal build test environment. If they don’t encounter any issues that a so severe they wrangle an IBM hotfix, they might move that on to early adopter users. And if that goes through, the resulting code stream might ultimately be deployed throughout the organization over the next two or three years.

So what the hell is actually being ‘supported’ here? Certainly not the end users, who are looking at around 8-10 years between having a problem and actually USING a solution. And that’s assuming the fix passes all the economic and engineering feasibility criteria. Given that almost 20 years after Lotusscript was introduced into Notes, users still get ugly popup boxes that say “Object variable not set. OK” when Jim from accounting forgets to null-check his NotesView object in the Approve button, I’d say that having official support from IBM is worth about nothing to the ultimate end users of the platform. Those people run to the Genius Bar at the local Apple store and buy iPads and subscribe to Basecamp and bypass IT departments altogether.

And who can blame them? They watch prime time TV commercials telling them “there’s an app for that.” They buy $99 discounted phones with access to 500,000 touchscreen, orientation-sensitive, location-aware apps that can take pictures of their faces and take voice input, then they go to work and use application templates in the Notes client that were built in the late 90s. It’s like getting to use a Star Trek transporter to go to the grocery store, but having to climb into a Model T for your commute to work.

It’s not like there aren’t solutions out there. They just aren’t ‘supported’ by IBM or by corporate IT departments. But that support means about as much to business outcome as a hood ornament means to a cross-country drive. The customers themselves will ultimately test everything before they roll it out anyway, no matter how many assurances they get from IBM. Of course, the testing process will be incredibly expensive too, since IBM taxes good governance practices by charging for every development and QA server the customer deploys, making properly segregated staging environments cost vastly more than they need to. Note that there is no “development” or “testing” configuration available for a Domino server on install that could cause it to mirror the behavior of whatever base server already exists in the domain. (I’d publish an open source solution to this problem, but then people would just say they couldn’t use it, right?)

The kicker is: I’m quite sure that Connectria and Turtle Partnership customers don’t call IBM support for anything important. They call Chris or Gab, because Chris and Gab are incredibly talented and hardworking people who are committed to the success of their customers. Whether IBM supports some configuration of software or some 3rd party solution means exactly squat to their customers, who are assuredly far more concerned with what Chris and Gab can do to achieve better business outcomes than what IBM manages to squeeze through the financial-and-legal-barriers over multiple years to maybe solve a problem if it happens to suit their EPS numbers.

So here’s my question to the esteemed Mr. Miller and Mrs. Davis, two of the most competent and devoted experts on IBM Collaboration technologies who have ever walked the earth: if your customers don’t use open source software because of IBM support concerns, why aren’t you offering to support your customers in their use of open source? Why are you letting them suffer through bad experiences on long-outdated technology when you know full well that there are world-class solutions available to them for free? Solutions that they can audit themselves; that they can report on directly to the individuals who wrote the code; that they can get fixes for in MINUTES instead of YEARS?

If you’re letting them suffer through this delusion that open source on OpenNTF is “nefarious”; when the best advice they can get in the world is “don’t do that because then the support intern from IBM won’t give you a useless PMR that doesn’t actually solve your problem,” well, it’s no wonder the platform is shrinking.

Posted in Uncategorized
3 comments on “Women are now allowed to choose their own husbands – I disagree with that idea
  1. Mike Woolsey says:

    Yeah, Notes designs has been changed by third parties at least back in Notes 2. Even the Notes client was hacked with some interesting tools. One of my colleagues added menu items to Notes to enable some of our third-party software as well. And Lotus released this thing called “the API” which gave us quite a bit of control over almost any database object, even to the point of creating nonsense and crashing things.

  2. Mike Woolsey says:

    I dunno about the advice you’re objecting to here — and I would agree with you, and more: there are far more stable design templates in the Open Source community than in the bought product.

    But I do have a comment.

    The issue I run into most often is, I can’t tell whether template X is one of those more-stable templates. I don’t have a way of distinguishing “this was a lousy choice” from “this was a terrific choice” for a different template. I just go in and look at the problem, and come out with a report, and then we pursue a solution.

    Now, I remember some horrific template software out there that was bought & paid for from Lotus. But it’s familiar. It’s memorable. We know what a bad hunk of crap those particular versions were. But: what we know less about, are less-used templates.

    Those templates I’ve had a chance to work with from openntf, I tell ya, I’ve no cause for concern. OpenLog is one of those things that should be in the product. IBM should pay someone for the privilege of such a great piece of code.

    But there’re a lot of things on openntf that I just don’t know enough about, to use.

    “We would rather keep the ills we have, than flee to others we know not of.”

    I’ve got in the habit of looking across the tabs of things in openntf.org. When I can see essentially a nice article on one guy’s experience in one of the screenshots or documentation tabs, then I’m comfortable enough to load that sucker up and see what it’s all about.

    There’s probably a pile of stuff I’m missing. But I haven’t trashed my installation trying to get stuff working that … well, it may be more work to figure it out than to do it myself with “the template we have, than flee to another that I know not of.”

  3. […] also prods me to finally write something I’ve wanted to publish for a while, going back to a conversation from last year that perpetuated a number of mistaken ideas about open source software. So here, from a […]

Take the red pill.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: