How do you balance rapid iteration and merging/upgrading? - eviltoast

More specifically, I’m thinking about two different modes of development for a library (private to the company) that’s already relied upon by other libraries and applications:

  1. Rapidly develop the library “in isolation” without being slowed down by keeping all of the users in sync. This causes more divergence and merge effort the longer you wait to upgrade users.
  2. Make all changes in lock-step with users, keeping everyone in sync for every change that is made. This will be slower and might result in wasted work if experimental changes are not successful.

As a side note: I believe these approaches are similar in spirit to the continuum of microservices vs monoliths.

Speaking from recent experience, I feel like I’m repeatedly finding that users of my library have built towers upon obsolete APIs, because there have been multiple phases of experimentation that necessitated large changes. So with each change, large amounts of code need to be rewritten.

I still think that approach #1 was justified during the early stages of the project, since I wanted to identify all of the design problems as quickly as possible through iteration. But as the API is getting closer to stabilization, I think I need to switch to mode #2.

How do you know when is the right time to switch? Are there any good strategies for avoiding painful upgrades?

  • tatterdemalion@programming.devOP
    1·
    1 year ago

    Thanks for your thoughtful reply.

    1. Yes. Yes.
    2. One or two major things. Breaking changes will usually result in a data structure format changing so algorithms that traverse the data structure need to be rewritten.
    3. One consumer is diligent about upgrading. The rest are much slower or rely on me to do it, but they continue building on top of an old version even after a new version is released.

    I like your idea of doing more frequent major releases and limiting the size of breaking changes within each release. It seems like a good compromise.

    • tinker_james@programming.dev
      2·
      1 year ago

      Hm. In that case, smaller more frequent breaking changes may also not be ideal. It sounds like no matter how small the breaking change, everyone who uses the library is going to have to update their code… and if it’s happening frequently, that could get annoying.

      This may be completely off-base, but just going off of what you said about data traversal, would it be completely out of scope for your library to provide a consistent interface for getting/traversing the data it is responsible for? Or do the consumers all use/traverse the returned data in very unique ways such that you couldn’t really develop a “general” API of sorts.

      • tatterdemalion@programming.devOP
        1·
        1 year ago

        would it be completely out of scope for your library to provide a consistent interface for getting/traversing the data it is responsible for?

        This is actually something I’ve been considering. I think it would make sense for me to see what existing traversals could be upstreamed into my library. Some of them might be very domain-specific, but others might be generic enough to be generally useful.