DSL might be the answer

A while ago I wrote a piece about the overuse of domain specific languages (DSL) in the software industry ¹, but soon after writing that post I found myself working on a DSL that simplifies the creation of a complicated build pipelines on Jenkins - DepBuilder. Does that make me a hypocrite or are the DSLs actually useful in certain cases?

Part I: the DSL is born

The DSL often starts its life in a meeting room where the HIPPOs (HIghest Paid PersOn) gather round the table to weigh the possible solutions to a specific business problem ². This problem could be anything from lowering the costs of the project to resolving a hairy technical problem that is hard to attack with the existing tools like deploying software to a bunch of servers.

They argue for a while, until a light bulb goes on in the middle aged tech lead’s head and they blurt: “What if we wrote a DSL?”

1. We will be able to move faster, since we will write a sharp tool optimized just for this problem.

2. The cost of the project will go down, since we could get the non-programmers to use it.

3. It’s an interesting technical challenge for our programmers.

4. (and I will be promoted for launching this project that will save my company a ton of money)

The tech lead starts sketching the DSL syntax on the nearest whiteboard: “Look how simple it is. We write some YAML and boom, we have the build pipeline. Or, we write some YAML and boom, the server is provisioned exactly how we envisioned. Imagine how many lines of Java code one would have to write to get to the same functionality.

The project manager, trying not to look stupid for not understanding the excitement in the tech lead’s voice, starts furiously scribbling in his little Moleskine notebook. “So, this looks like an interesting approach, but hmmm, how much time and personnel do you need to actually pull it off? We are already way over our budget for this project, so…”

“Well, it’s hard to say without knowing all the details about the problems the tool is supposed to solve, but about 6 months and 2 programmers sounds about right to me,” says the tech lead.

“It’s going to be hard to justify this additional expense to my boss, but… fine. It looks like you got it all under control. Well, we will circle back next week to finalize the details, anyone up for a lunch now?” wraps up the project manager.

Part II: early success

The team circles back and since the budget related things are looking good, the plan gets approved and two proven developers are assigned to start working on the project. It’s their first time writing a DSL, so they struggle for a few weeks but eventually the prototype is done. Their deployment tool passes their first smoke test, which really proves that the DSL makes deploying their software easier.

The tech lead is over the moon with the early prototype and confidently walks into the progress meeting, telling everyone that wants to listen about the great new tool that will revolutionize the way they deploy software at BigCorp.

After the meeting, the older one of the two developers brings the tech lead aside and starts talking with a raspy voice: “There is something you should know about this DSL, before we start promising the moon. The DSL has a couple of really rough edges, that are going to cut us at some point, unless we extend the looming deadline and sand them off.”

“Of course, of course. I entirely understand the problems you are having, I am a programmer myself after all. You will figure it out. I know you will. Been there done that, there is nothing to worry about. Keep me posted!” answers the tech lead. The project manager is also pleased with the progress, and the project goes on its merry way.

Part III: we need more functionality

When you are working on the same project for a long time, you eventually realize that the problem is way harder than what you have anticipated in the beginning. It brings similar feelings as buying a used car does. For the first few days, the car works fine and you are happy with the purchase (the prototype looks promising). With time, an unexpected problems start popping up (features and edge cases), until you realize that everything is broken and you swear to never buy a used car again (move on to a different project hoping to never touch the previous project again). But, let’s go back to our little story.

The project progresses fine and the developers are sitting in their cave, enjoying in sounds of their mechanical keyboards. “Click, clack,” says the keyboard. “We need another feature,” says the tech lead.

“We have to allow our users some kind of a control flow. Say, I want to deploy our software, if and only if, the server is numbered with an odd number.”

“Uh, oh! This DSL wasn’t really designed to support this from the start,” groans the older of the two developers. The younger developer wiggles with ears: “Actually, we could add a rudimentary for loop and an if statement. It’s going to be very primitive, but it’s not impossible to add this feature.”

“Great, yeah let’s do that. I am glad you two have it under control. I have another meeting right now, but either way, keep up the good work,” the tech lead storms out of the room like a hurricane, unwilling to take the no for an answer.

The doors haven’t even closed yet, when the older developer mumbles in his beard:
“For loop in a YAML file? What?”

Part IV: the DSL strikes back (or everything is on fire)

What is the point of a domain specific language, if it solves only a very specific problem in that domain? Wouldn’t it be better if we could use it to solve every problem in that domain ³? A programmer in me says: “No. Small tools yadda, yadda, Unix philosophy, yak, yak, yak.” An executive in me says: “What are you even talking about? More features is always better than less features. If the DSL can cure cancer on top of deploying software, I’ll take it.”

Another department starts using the DSL for deploying their production services. They keep finding themselves in the corner and are mercilessly posting feature requests based on their production needs. The DSL team now consists of 10 full time developers and things are getting kind of out of hand due to all the unexpected features that don’t really fit into the rest of the language. There is no turning back now, it’s an important project for the company that will be finished no matter what.

The DSL itself has become a very powerful language. It has a nice logo, a catchy name, the spec is at 450 pages and it almost feels like a regular interpreted language, except it grew in an ad-hoc fashion and is full of weird quirks. The original creators would be proud of it. Shame they left years ago.

The project manager calls an emergency meeting with all the stakeholders present: “Fellas, we have to do something about our deployment language. The costs of our project are completely out of control, we have a backlog of 2000 bugs, there is a never ending stream of complaints from the system administrators that are forced to use this quirky language for managing their fleet of servers, there is…”

A bright tech lead in their early thirties stops him in mid sentence: “What if we wrote a DSL?”

But wait, the title of this article claims otherwise

Ah yes, I almost forgot about that. I believe the DSL becomes useful when you have a small, well defined problem with known predefined constraints that are not going to change. The DSL user also shouldn’t be a total control freak that cares about the implementation details of the DSL. In other words, the user should care about the end result and not how to get to the end result ⁴.

After spending quite some time setting up the build pipelines for a large project that consisted of many dependencies, I started to believe that the DSL is the right solution for this type of problem. As the end user, you mostly don’t care how your dependencies are being built, as long as they are built in the right order and on the desired build agents (when you are trying to build the same code on multiple platforms).

The problem itself is well understood and apart from defining how to how to handle dependency build failures (do you abort the build or continue building in case of a build error) there is not much more to it. The rest of the complexity should be handled by the DSL implementation, which is also how the DepBuilder handles the pipeline builds:

// yes, it's really that simple
TimeLib -> Server -> "Build Installer"
TimeLib -> MobileApp -> "Build Installer"
TimeLib -> DesktopApp -> "Build Installer"

The DSL above, will create the following diamond dependency build pipeline out of the existing Jenkins jobs (see also Dependency management with Jenkins ):

When running the build, the dependencies that could be run in parallel are automatically built in parallel, without the user losing their mind while trying to figure which parts of a large pipeline could be parallelized. This is the true power of the DSL ⁵.

What often happens in the real world, is that somebody will realize that the DSL is not covering a certain edge case and will try to bolt additional functionality on top (like for loops). While in theory you can keep adding features indefinitely, at some point you have to ask yourself: “Do we really want to handle this edge case or we are better off by not doing that?” At that point you are essentially building a crappy programming language that is slow, has inconsistent syntax and is full of weird behavior with no proper debugger support.

Notes