Re: AreaDetector repository inconsistent

["J. Lewis Muir" <jlmuir@imca-cat.org>]

On 02/05, Mark Rivers wrote:
> Hi Lewis,
> 
> > I don't understand why you want to separate out into multiple
> > repositories things that everyone will just turn around and combine
> > again.  Why not just leave them together?
> 
> Here are a number of reasons:
> 
> - Clear separation.  When we release a new version of ADPilatus people
> who don't use ADPilatus but do use ADPointGrey know immediately that
> this change does not affect them.  If the Pilatus and Point Grey code
> were both in a large areaDetector repository then it would not be
> obvious.  Currently Point Grey users only need to watch for releases
> of ADCore and ADPointGrey.  If it were combined they would need to
> watch for every release of areaDetector.  Since there are now 40
> detector repositories in areDetector, to first order they would need
> to read the release notes 20x more often.
>
> - Easier collaboration.  With separate repositories we can assign
> permissions differently for each one.  I could give you write access
> to ADADSC since you wrote it, but not give you write permission to
> ADCore.
>
> - Lower barriers for new detectors.  Because each detector is in a
> separate repository the barrier to accepting a new detector is lower.
> They are well decoupled from ADCore. If ADxxx it does not work with
> some change to ADCore it does not impact the release of ADCore, and
> ADxxx will still work for earlier releases of ADCore.  We don't have
> to wait for that developer to fix it.
 
Hi, Mark!

Thanks for sharing those reasons; they all make sense to me.

> Consider the analogy with synApps.  That is about 25 modules which are
> in separate repositories (motor, calc, etc.) and need to be assembled
> into a tree like areaDetector.  Are you suggesting that they should
> also be in a single repository?

Yes, I would love that! :-)

We obviously differ here, and I do understand your reasons given above,
but I suspect a more fundamental reason we differ here is because of
how the modules included in synApps (and synApps itself) are developed
compared to how, IMO, they could be developed.  What I see with synApps
is all (well, most of) the modules I want.  And just like areaDetector,
I might not use them all, but it's convenient to just get them all and
build them all (or configure the build to include or exclude some).
synApps is great in that it has provided a way to get all those modules
at once and working correctly together.  That's a huge plus.  However,
synApps releases happen so infrequently that I essentially have to make
my own internal synApps releases which takes a nontrivial amount of
time.  I would much rather have a synApps (and the modules it includes)
that adheres to semantic versioning <https://semver.org/>, does not
bump its major version every release, and releases often so that I can
get (backward-compatible) features and bug fixes without having to read
every entry in every module's changelog (if it has one).  I don't want
to have to spend time dealing with breaking changes (i.e., changes that
are not backward compatible) for every release.

Maybe a good analogy to this would be if I wanted to run RHEL 7 on all
my machines for five years, but instead I had to upgrade to the next
major release (e.g., RHEL 8, 9, 10, etc.) every three months.  I really
value the API stability of a RHEL release and don't want to have to
spend time changing and upgrading my software stack to work with each
new RHEL release every three months.

> The synApps repositories get updated regularly, but the parent synApps
> release has only happened twice in the last 5 years (5-7 and 5-8).  I
> think that is an indication of how unbundling improves the frequency
> of releases.

Right, and with the way the synApps modules are currently developed,
I agree with you.  However, I would bet that the reason synApps does
not get updated regularly is that it's a lot of work to do it.  Part of
the reason it's a lot of work is all the breaking change.  If all the
modules in synApps adhered to semantic versioning and did not introduce
breaking changes left and right, making a new synApps release would be
significantly easier.

Obviously, though, making backward-compatibility a priority would have
to be something that the community really wants.  While I and maybe some
others (others, are you out there?) might wish for it, if the community
doesn't want it or the developers don't want it, then it's not going
to happen.  Also, I understand that there can easily be very different
philosophies and concerns/needs here (e.g., one synchrotron beamline
adds or changes hardware all the time vs. another that builds out a
configuration for one type of experiment and doesn't change the hardware
much).

So, I understand your three reasons given in support of a
multiple-repository approach, and they all make sense.  Thanks again for
that!  I do think that the first one about clear separation and reading
the release notes would be much less of an issue if the development
approach was different.  Again, to draw an analogy with RHEL, a new
minor release could include kernel drivers to support new hardware, and
I don't mind because it doesn't break backward-compatibility.  I just
install the latest updates for RHEL 7 (if I'm running 7), and I don't
need to read release notes.

Regards,

Lewis