Reconsider dependency on zope

[edmorley]

Hi!

Newer minor versions of the zope.interface and zope.component dependencies now pull in many more transient dependencies than they used to:
https://github.com/mozilla/treeherder/blob/1bbc11c09b36c1aeb642fe8057647bc7c60c796d/requirements/dev.txt#L119-L184

This feels pretty heavyweight for something that doesn't seem entirely necessary. (And is adding to our dependency bot PR churn, plus on multiple occasions needed manual changes to add the hashes for sub-dependencies).

Could the zope dependencies be removed?

[davehunt]

Thanks for raising this @edmorley. Which dependency bot are you using? If you're using Dependabot, it has an option (I believe it's the default now) to only update direct dependencies, unless a transient dependency has a security update. This might reduce your PR churn, as it shouldn't be necessary to keep zope and it's dependencies up to date.

To answer your question though, I'm sure we could revisit if zope is needed, and I'd be happy to review and merge a pull request that removes this dependency. Perhaps @davidemoro has some additional thoughts on this.

[davidemoro]

Hi @davehunt, @edmorley,

let me add a little bit of context about those zope.* packages.

First of all we didn't introduced the zope dependency (zope as the big fat web framework) but a couple of battle proven libraries that should be considered as an internal implementation detail:

• zope.interface
• zope.component

used by the zope web framework and many other web and not web based technologies.
Basically the zope.interface introduces the interface concept while the zope.component the lookup machinery that let you get the right adapter.

Unfortunately the PR churn downside is caused by the zope publishing modular approach (ship many small packages instead of following a monolithic approach).

Obviously all dependencies can be removed or replaced but that means reimplementing what you are going to drop on PyPOM side with different pros and cons:

• implement, test and maintain on PyPOM side a custom more lightweight pluggable system
• time needed for developing a lightweight solution on PyPOM side
• more/less PR churn
• more or less flexibility managing external dependencies
• sometimes when a major release happens you have to update 2 dependencies at the same time (for example pytest and pluggy requiring both a new version. If you upgrade them singularly your PR will fail, you'll have to upgrade both components)

So let's evaluate carefully the pros and cons of both solutions and let's think together the next steps for PyPOM. What I take care is not using a dependency or another but a page object library that supports both plain Selenium, Splinter or any other future engine.

I hope my comment was useful!

[edmorley]

Thank you for the detailed replies :-)

Which dependency bot are you using?

We're using pyup for Python deps (and Renovate for JS), but considering switching Python dep handling to Dependabot given pyup lack of activity. (Ideally we'd use Renovate for everything, but its Python support doesn't handle hashes at the moment I don't think. It also doesn't yet support pipenv or Poetry)

If you're using Dependabot, it has an option (I believe it's the default now) to only update direct dependencies, unless a transient dependency has a security update. This might reduce your PR churn, as it shouldn't be necessary to keep zope and it's dependencies up to date.

The thing is that we do want to update most transient dependencies. The reasons doing so is painful at the moment are:

• using requirements files with hashes means manually adding/removing transient dependencies (pyup can keep them up to date once added, but doesn't know when to add/remove them)
• not having of an equivalent of a JS lockfile means we can't reduce the PR noise from transient dependencies by having a single weekly "lockfile refresh PR" (example: Lock file maintenance treeherder#4202)

Now pipfile.lock would solve much of this, but given it's not supported by pip natively yet, that means using pipenv and all the problems that come with it (slow performance, many regressions both locally and with the Heroku integration; more background).

I'd like to check out Poetry at some point, since it's apparently much faster than pipenv, and is also supported by Dependabot:
https://dependabot.com/blog/announcing-poetry-support/

First of all we didn't introduced the zope dependency (zope as the big fat web framework) but a couple of battle proven libraries that should be considered as an internal implementation detail
...
What I take care is not using a dependency or another but a page object library that supports both plain Selenium, Splinter or any other future engine.

Yeah I realise it's not all of Zope the web framework, and that it's an internal implementation detail.

I guess my main point is that "all" I would like to do is run some selenium tests with pytest. I have no need to use another engine/pluggable architecture. Ideally I would just install pytest-selenium and it would offer functionality for the 80% use-case, and I would rarely need to install any other packages.

If there is a genuine need for a pluggable architecture in PyPOM, then fair enough - but given that splinter is presumably less relevant in a world of Selenium/webdriver/browsers supporting headless natively, I wonder if that's still the case?

[davehunt]

@davidemoro splinter is already an optional extra for PyPOM.. I wonder if we could make the zope packages also optional so they're only installed/used when splinter is requested.

[davidemoro]

Hi, I got your point and since the probability of the introduction of new selenium engines is very low, we can drop all the zope.* dependencies and use a very simple check based on the class of the engine with few code base changes and low impact.

…

On Wed, 31 Oct 2018, 18:20 Dave Hunt, ***@***.***> wrote: @davidemoro <https://github.com/davidemoro> splinter is already an optional extra for PyPOM.. I wonder if we could make the zope packages also optional so they're only installed/used when splinter is requested. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#95 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAlIqhdkRbmKPXXib8fSXocyAYD3yI6Yks5uqdv6gaJpZM4X--O6> .

[davidemoro]

But I still do recommend using existing packages vs a rewrite