[nvaytet]

Instead of trying one server 3 times and then moving on to the fallback, we switch to fallback on first failure.

[jokasimr]

Why is it preferable to switch to the fallback earlier?

[nvaytet]

Because if the dmsc server is down, we might as well try the other one instead of waiting for the 3 failures on the dmsc one.
On visa for example (although it seems they may have fixed it now), each attempt on the dmsc server took a minute or so. So you had to wait 3-4 minutes where nothing was happening before switching to github and succeeding.

Now, we try once, and if it fails, go to GH. If that fails, go back and try dmsc a second time. I think it makes sense, no?

[jokasimr]

The issue on visa should be fixed now. If it was the http_proxy issue. Brian told me today it's been fixed.

I haven't seen any file server timeouts in the CI lately, not since DST wrote that they figured out that issue where requests were routed wrong in the network somehow.

I think the changes here are fine, it just seems simpler to use the built in retry mechanism.
Would it make sense to put the GitHub fallback in front of the file server in the list of sources?

[nvaytet]

I think the changes here are fine, it just seems simpler to use the built in retry mechanism.

Yes it would have been if it didn't take something like a minute per attempt (or whatever timeout limit is enforced).

Would it make sense to put the GitHub fallback in front of the file server in the list of sources?

We didn't want to hammer github every time we run CI or nightly tests because some DMSC/ESS ip addresses have already been blocked by github (we can't pip install from a gh repo on the nightly test gitlab runner because of this issue).

[jokasimr]

I see, okay that makes sense 👍