Integration as a Separate Responsibility in Software
The whole is more than the sum of its parts
In the world of software structure, things are bound to go awry. The components don't always click together perfectly. Instead of the duct tape approach to programming, where things are loosely stuck together for quick fixes, we're often guilty of over-engineering — welding parts as if change is never going to come knocking. This approach is a tragedy in the making and, let's be real, it costs a fortune. Why? Because welded joints are a pain to adjust; they're stiff, and when the time comes for tweaks or tests, it's a whole production.
So, what's the deal with this 'welding'? And what's the other option?
You've got 'welded' parts wherever functions are so tightly codependent that they're practically conjoined twins. Functions across the same or different modules are constantly high-fiving each other and doing a bunch of extra stuff on the side.
Functions get things done through logic. They run the show and dictate software behavior. But as functions grow up, they start to outsource their logic to keep things tidy or borrow from the logic of other functions that have already figured stuff out. This is how functions that are supposed to be creating behavior with logic end up in a loop of dependency on others.
This might seem like standard practice — but don't be fooled, it's far from ideal. Writing logic within a function feels effortless at the moment, but later down the line, it turns into a nightmare for clarity and testability.
How did we even get here?
Division of Labor in Organizations
Well, it all starts with how we've traditionally thought about division of labor, especially within organizations. Programming took a few pages from the organizational playbook, not engineering. It's all about command-and-control.
Picture the evolution of a startup:
Initially, it's a one-man show, the founder, juggling everything.
Enter the first division of labor: employee number one. The founder starts delegating, and the workload lightens.
More divisions follow. Both the founder and the first employee start offloading work to more team members.
Here's the gist: when someone's swamped, they pass off duties. Instead of getting their hands dirty, they start directing traffic and checking the results that matter for their work. Plus they are still working on addition tasks themselves.
Let's zoom in on the founder-employee dynamic:
The founder's to-do list shrinks from five to two areas of work, offloading three to the employee. But now there's the added job of managing — dishing out tasks, checking on results, and integrating those results into their own work. That's the new black box of responsibility.
This is pretty much how companies roll.
Division of Labor in Software
Now, let's talk software:
It all kicks off with a big, chunky block of logic — the beating heart of your software.
When this chunk gets too bulky to handle, it's time to split it up: part of the logic gets a new home in another function.
As the logic keeps ballooning, the cycle repeats.
A Code Example
Need a real-world example?
Take the development of a word guessing game app. The user tries to guess a word from a list, and the app clues them in on the letters they've nailed.
Initially, one function holds all the cards — the entire application logic. Through a couple of refactoring rounds, this logic gets divvied up among multiple functions.
Each time you peel off some logic into a new function, you leave behind a trail of logic in the original function, now sprinkled with calls to other functions. These are the infamous functional dependencies, making the remaining logic in the calling functions a tough nut to crack when it comes to testing.
So, how should we be splitting tasks in software?
Clean Division of Labor in Software
Well, the current state of affairs in both organizations and software is a bit of a mess. Things somehow work, but they're anything but neat, comprehensible, or easy to adapt. We're missing a trick by not taking seriously the old saying "The whole is more than the sum of its parts".
Let me break it down with some simple math: Picture a whole number, say a 2.
We all know that 2 equals 1 plus 1, so here are the constituent parts.
But what makes the whole greater than the sum of its parts? It's all about how the parts are arranged to actually be parts and not something just “floating around in space”. They need to be assembled. The sum of the parts is just their mere presence; the whole is this sum plus the way the parts are put together.
Think of composition as the magic that turns parts into a whole. It's an essential step in crafting something bigger and better from the individual pieces.
Yet, this magic is often missing in action, both in companies and software development. We fail to see composition as a separate, crucial ingredient; but it’s an activity and responsibility all of its own. The result? Overwhelmed employees and overloaded functions. Everyone's expected to do their thing and also pull the whole ensemble together. It's like mixing up responsibilities, violating the Single Responsibility Principle (SRP), and ending up with a function that's all over the place in terms of abstraction, defying the Single Level of Abstraction Principle (SLA).
What does a clean division of labor look like in software, then?
It's all about differentiation that leads to diverse functionalities plus an explicit composition of these into a coherent whole.
Again, we start with a whole that contains all the logic needed to drive the application.
When we break this logic into separate functions, we introduce another player — a function that's all about composition, with zero logic of its own. Its sole job is to blend the performances of the other two functions.
And as we continue to dissect the logic, new functions emerge to handle the composition of the increasingly granular parts.
These compositional functions act like glue, binding logic-laden functions into a cohesive unit.
I like to call this process integration. It's like fitting puzzle pieces together to complete the picture.
A Code Example
How about another code example?
This "n+1 decomposition" technique — creating n parts plus one more for their composition — feels straightforward and organic. It's like how we've always built machines and electronic circuits: there are the functional parts, and then there are connectors. In electronics, think circuit boards.
But hey, after decades of software development heading in a different direction, it might be hard to wrap your head around this. Does this strategy really work for code? Can we separate functionality from composition?
Absolutely, yes. And it's a game-changer for both testability and understandability.
Let's revisit the word guessing game app. This time, I'm not just refactoring; I'm showing you a design that separates logic-rich functions from integration right from the get-go. Then, I'll present that design turned into code.
The Design
It all starts with a bird's-eye view of the design: an application that's there to serve the user, packed with all the logic needed for that purpose.
The user relies on the application.
But we're not building a monolithic beast; we're aiming for testable, understandable parts.
Right from the start can spot two distinct parts: the user interaction logic (the Portal or Frontend) and the game logic (the Domain), which doesn't care about the user interface.
Given what we've discussed, it's clear that these two parts shouldn't be directly fused. They need a third element to tie them together.
This third element, the composition, essentially 'wraps around' the parts. It's often implied, but it's much better to make it explicit, especially when it comes to coding.
For implementation, though, that's not quite enough. We need to pin down what these parts should do and when. To do that, I adopt an outside-in approach, focusing on what the user expects from the application, as shown in their interactions with the portal.
I see two key interactions:
The user kicks off the game.
The user enters a guess, and the app judges whether it's correct.
At first glance, this interaction diagram might seem to feature only one logic-bearing part, the Portal. But it hints at a second: a Processor, which handles the requests from the Portal. And, naturally, there must be a composition wrapping around both.
Keeping the user interface as one big chunk doesn't sound like a good move. Gathering user inputs (Collect) and displaying evaluation results (Project) are quite different tasks. So I split the portal into two and make the composition around all three pieces clear as day.
Now let’s zoom in on the Processor:
The first request launches the game at program start.
The second request is a repeated attempt to guess the word.
The first function unit is a no-brainer — it generates a word to be guessed.
The real meat, though, is in that second function unit: the guess evaluation. That's where the domain logic flexes its muscles. To really nail this part, let's break it down further:
Firstly, check if each letter in the guess word appears in the solution word at all.
Then, for those that do, check if they're also in the correct position.
Notice how the original "Guess game," which was all about logic, has now transformed into an integration sans logic? The logic has migrated into the separate parts below, leaving behind the need and responsibility to continue weaving these parts into a whole.
All the pieces are prepped. Time to assemble the application.
But upon closer inspection, there's room for more composition. The loop of user input and result display feels like it needs its own bracket. I want to highlight this with another layer of integration:
Here's the takeaway: Integrations, the 'glue' between parts, can be added retroactively. The new wholes that result are easier to grasp than if we were constantly fixated on the parts. They also become standalone units ripe for testing.
This final design showcases multiple layers of integration. Climbing from the bottom up, each layer takes the level of abstraction up a notch, a concept known as Stratified Design.
Each layer is essentially a language with its own unique vocabulary, becoming more abstract as you move up. Can you spot the languages?
At the top, the language is super abstract, with just one word: "Run". "Run" encapsulates the entire functionality of the application. It's abstract yet highly specific to our word guessing game.
The layer below is a tad less abstract, with a vocabulary of "Start game" and "Play".
Dive deeper, and you hit the language focused on the nitty-gritty of gameplay: "Collect", "Project", "Check guess", and "Start game".
And the most tangible language includes "Start game", "Check right letter", "Check right position", "Collect", and "Project".
On a yet lower, not visible stratum lies the programming language, the nuts and bolts of logic. But it's so concrete and universal that it's not suitable for design. Design is about rising above that level of concreteness. That's why designing is the art of crafting new languages during programming, making it easier to describe the processes that breathe life into the application. The best designs evolve gradually, creating a hierarchy of languages that layer the solution.
Each language can articulate the entire solution. Depending on your grasp, you select the language that resonates with you. Sometimes you want the fine details; other times, a broad stroke will do.
The Code
This design is a breeze to translate into code. To keep it simple, let's just focus on encapsulating logic. Each shape in the design becomes a function. Some are all about logic; others are dedicated to integration.
Composition happens on two fronts:
Merging functions into a new entity is what I call integration.
Gathering logic into a new entity is also a form of composition; the functions born from this process are operations.
In the broader sense of Stratified Design,
composition is about connecting elements of one language to form a building block of a higher-level language.
While turning the design into code, I hit a few snags. Not everything matched up perfectly with the flow in the design. But hey, that's not a dealbreaker. It's part and parcel of the process.
Design is the dream, coding is reality. Things might look peachy in theory, but they may not translate directly into practice. Yet, the dream isn't a waste. At the abstract level — visually and in higher languages — solutions come more easily. Sometimes you go top-down, sometimes bottom-up. That's just the way it goes.
The functions in the code match what the design shows. What's worth mentioning is how I tackled integration in "Play". The design shows that "Collect" produces a guess word that gets evaluated and then displayed ("Project"). But this repeats several times. How does that work without a visible loop in the design?
Here's the secret: the "*" in "(guess:string)*". That little star means "Collect" delivers multiple outputs. It's a data stream.
I've used a continuation (a function pointer) to handle this stream. So, in the code, you'll see a lambda expression as the second parameter of "Collect". That's a nifty coding detail. Don't sweat it right now; focus on the big picture.
But mind you: this principle isn't just for functions; it extends to modules. In the code, you'll see hints of modules marked by comments. Let me spell it out for you:
And that brings us full circle, back to the start of the design. It all begins with modules as the rough building blocks of functionality. Then we refine with functions to finely tune that functionality.
Only functions can carry logic and call upon other functions to forge behavior. That's what composition is all about.
Modules are like neat little piles, grouping functions to make sense of the chaos. That's aggregation.
Yet, modules can also have a focus, driven by the functions they encompass. They might lean more towards operations (like "Portal", "Domain") or towards integration (like "Application").
The Principle
Let me leave you with the core principle that's absolutely vital for software development. Considering we're dealing with some pretty complex "machines," we can't just slap them together. We need to envision and assemble deep hierarchies of building blocks being wholes consisting of many parts being wholes themselves and consisting of more parts themselves and so on. For this, the golden rule is:
When you break down a functional unit into n parts, always add 1 more to the mix — the one that'll keep the original essence intact. This extra piece knows all the parts and weaves them into a coherent whole. It’s the composition.
This idea is even more foundational than DRY or SRP. DRY and SRP are all about content; this principle is about the fundamental shape of software.
I've referred to it as the IOSP (Integration Operation Segregation Principle) elsewhere. But, you could also think of it as the NP1 or the N Plus 1 Principle.
In other words: The whole is indeed greater than the sum of its parts.