On Complexity

Recently, I was having a conversation with a smart co-worker.  During this chat, it was revealed that he felt a new system that I was building was unnecessarily complex.  Without disagreement, I asked for further explanation why he believed this to be true.  After some back and forth, he said that all the technologies being put in play were unfamiliar, and that is what makes it complex.

My immediate response was, "Just because something is unfamiliar does not make it complex."  And I believe that to be true.  

Unfamiliar tools may be very simple to understand and use, like a box clamp for joining perfectly square wooden boxes.  It's very simple, yet at first glance it may not be entirely evident how to use it.  Seeing it in use tells the whole story, though.

Other tools may perform very complex tasks, but their usage can be very simple. An example is the Automated Teller Machine, which performs a surprising number of complex tasks without requiring much of the user whatsoever.  To most people, this is a very familiar tool, and while it actually is complex, we don't think of it as such, because it is also understood.

After considerable reflection, I think the word complexity is not always applied to a thing to describe the thing.  Sometimes, complexity is used to describe the effort of understanding a thing.  There is a cognitive tax that must be paid to learn something well enough to be able to use it effectively.  Sometimes there is time and effort available to acquire that knowledge.  Sometimes there isn't.  And with nearly every transaction you make in life, there's an opportunity cost you must also consider--part of the cost learning this new thing is whatever you might've done instead, and that is now lost as well (or at least delayed).

With any new system, there is much to learn and thus it all feels very complex in the beginning.  Study is required.  Errors are committed and mistakes happen.  The mental tax is paid, understanding is obtained.  Once these new tools have become familiar, the apparent complexity melts away and then they are just tools in your back pocket.

A fair question to ask, though, is "What is the value proposition of this new and unfamiliar system?"  Excitement due to the newness of something is not sufficient to deploy it.  This is how server systems end up with Python, C#, Java, Golang, Redis, Mongo, and MySQL all Frankensteined together to handle a handful of simple tasks.  No, there has to be some reckoning of long term value, that the mental tax for teaching a developer the stack is not only worth paying, but pays you back with accelerated development in the future.

A great architect should be able to meet the current needs, recognize the near-term requirements, and anticipate the likeliest future requirements simultaneously, and lay the groundwork today by selecting the technologies that dovetail nicely in a way that enables the best future outcomes with minimal future effort.  Sometimes that comes with a high startup cost.  So long as the balance is ultimately in your favor, it's a win.