I have fixed gradle and updated it to Bullseye.

I think the last step is to remove RT from TranslateWiki.

https://gerrit-review.googlesource.com/c/plugins/download-commands/+/359823 got merged. The plugin is bundled in Gerrit, so I guess that will be included in the next versions of Gerrit 3.5, 3.6 and 3.7.

I have merged all changes, Marked the repository read-only and deleted the GitHub mirror.

The instance can stay down for now, I will only need it to test the Gerrit 3.5 > 3.6 upgrade. I will address the authentication issue when I start preparing the Gerrit upgrade (at some point over the next few weeks).

I have confirmed the fix via https://gerrit.wikimedia.org/r/c/integration/quibble/+/897884/ which depended on the pending change for Parsoid. Thank you @cscott for the super quick fix on a Monday morning :)

In T330960#8679878, @LSobanski wrote:

@hashar A follow up question, is there a plan to configure releases2002 so that the switchover is possible? I'm asking both in terms of testing this within the DC switchover period and longer term operational expectations.

I went to do a diff between there two builds, they both have parsoid at 26a03541b821deb8746da87a4db32acdde289039

Solved by reinstauring rules https://gerrit.wikimedia.org/r/c/operations/puppet/+/887363/ and some further refactoring via https://gerrit.wikimedia.org/r/c/operations/puppet/+/887738/

The scap fix has been done in T330394. For production we still have to update git fleet wide and some bits have been done with @MoritzMuehlenhoff

(you could name it DUCT) that is an excellent robot name, then aw-gitlab-bot is fine as well, it is merely an implementation detail.

I'd rather share the credentials and use a single account for that robot. That sounds easier to have only one account reporting to Gerrit which I think will make it easier to lookup the robot doc, find the point of contact if there is an issue etc. It also simplify adding another instance since you don't need to create a new account or having to add it to Gerrit Service Users group.

I have merged the archiving change https://gerrit.wikimedia.org/r/c/mediawiki/extensions/BlueSpiceBookshelfUI/+/895743
Deleted the GitHub repository
And made it read only again

I have made it writable again so the full archival can be completed. Thanks @Kizule!

In T331626#8680354, @hashar wrote:

PROCS CRITICAL: 13 processes with UID = 38 (list), regex args '/usr/lib/mailman3/bin/runner'
Last State Change: 2023-03-07 14:37:21
Last Notification: 2023-03-07 14:43:18 (notification 1)

Mail should be emitted again, it will take a bit of time to clear the queue though. That can be monitored via https://grafana.wikimedia.org/d/GvuAmuuGk/mailman3

Icinga says OK: mailman3 queues are below the limits, but there is an alert about the runners:

I am guessing it is an issue with Mailman. https://grafana.wikimedia.org/d/GvuAmuuGk/mailman3 shows a large queue since March 7th 14:12:

I know nothing about the mediawiki-commits, in mailman3 it doesn't show up at all and the archive show 0 discussions at https://lists.wikimedia.org/hyperkitty/list/mediawiki-commits@lists.wikimedia.org/

After four years, it is better to file a new task since the root cause is most probably different :)

The immediate issue is solved, the selenium-daily jobs are back to NodeJS 14.

This is because we use wdio-sync which depends on fibers https://www.npmjs.com/package/fibers-nodejs which is no more supported https://github.com/laverdet/node-fibers#readme

Awesome thank you!

I have confirmed it works. I am reopening so that the Datacenter-Switchover documentation gets updated to take in account the keyholder-proxy.service has to be restarted to take in account the latest config. I guess I can pair that with @claime :)

Part of the fix for the deployment servers and scap is https://gerrit.wikimedia.org/r/c/operations/puppet/+/891555 by @nfraison which is to have Puppet to run the git command as the deployment user rather than root. T330394#8640706 as details about implementing and testing the fix.

@cjming thank you for the very well crafted issue report! @elukey mentioned the same issue earlier today and I believe the revert has fixed it. I did a recheck on https://gerrit.wikimedia.org/r/c/mediawiki/libs/metrics-platform/+/892628 and apparently it is running properly now.

+ ci-test-error which is where we track anything that happens to fail on CI for whatever reason :)

I have been confused by the group mismatching and the files permission/umask. Here is a summary of my investigation.

Nice. trebuchet is the previous deployment system which has been entirely superseded by scap more than a few years ago. I guess we kept the name around to give some preliminary UID to the file when Puppet runs the repository initialization. It is worse fixing for sure (maybe use mwdeploy instead?) but I don't think it is related to the issue we are encountering.

In T325128#8667625, @MatthewVernon wrote:

@hashar are there still things that need doing on this task? It looks like
https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ is now clear.

[asking because this is on the clinic duty dashboard]

Doing further digging, all the files which are not group writable under /srv/mediawiki-staging are in the wikidev group. I think they should be in the deployment group instead.

I have removed the git staged files from /srv/mediawiki-staging restoring its state.

Things I have checked we do have the profile script to set umask:

# this file is managed by puppet
# set umask to 0002 for wikidev users
# to prevent broken repos per T79400
if groups | grep -w -q wikidev; then
  umask 0002
else
  umask 0022
fi

At first I have suspected a rsync issue when we switched over cause some files are owned by UID 498 which has no name associated with it. There are plenty of objects owned by deployers which are not group writable since Thursday March 2nd 9:05.

That is the first time the train runs since we did the datacenter switch over. The steps used to move the deployment server are listed at T330651

I have created the Gerrit repo mediawiki/services/machinetranslation https://gerrit.wikimedia.org/r/admin/repos/mediawiki/services/machinetranslation,general

I have upgraded both Gerrit instances and confirmed they have the whole set of plugins enabled (and at 3.5.5).

I have upgraded Gerrit from 3.5.4 to 3.5.5 which upgrade sshd to 2.9.2. There are some changelog entries, one mentions ed25519 and ssh-rsa signature:

• SSHD-1231 Public key authentication: wrong signature algorithm used (ed25519 key with ssh-rsa signature)

But I don't think it exactly matches our task. The ssh key extensions will be released in Gerrit 3.6 iirc.

I have deleted some old replication tasks that were left over on the server since August 2022:

14:00 	<hashar> 	gerrit: removed old replication tasks from /var/lib/gerrit2/review_site/data/replication/ref-updates/waiting (most from August 5th 2022)

When browsing the Check UI that looks like:

https://releases-jenkins.wikimedia.org/ is back on releases1002 which has the fully configured Jenkins instance.

Well that can still be explored :-]

I would like to investigate having Gerrit to fetch the output from the Jenkins build and insert the result to the Check tab. From the Check API I think that is CheckResult.message field:

/**
 * Exhaustive optional message describing the check result.
 * Will be initially collapsed. Might potentially be very long, e.g. a log of
 * MB size. The UI is not limiting this. Data providing plugins are
 * responsible for not killing the browser. :-)
 *
 * For now this is just a plain unformatted string. The only formatting
 * applied is the one that Gerrit also applies to human comments. TBD: Both
 * human comments and check result messages should get richer formatting
 * options.
 */
message?: string;

Which would require Quibble to generate a json file containing the details of the failure, we could then insert it in the UI. That might be more robust than using grep.

From my mail earlier today, that broke the release Jenkins service.

Marking solved as far as we are concerned. Upstream will eventually find a proper fix. Please reopen if you still see ssh connection errors and I will dig again ;)

I do not see any occurence of SSH_MSG_CHANNEL_WINDOW_ADJUST or unassigned in the Gerrit error_log files. So I guess that kind of fixed it! Clark from OpenDev talked about that issue during the monthly Gerrit community meeting ( https://docs.google.com/document/d/1QmrIsyBx52Sk_qWwrcci0hI_SlUWMGiwT8Kp05_ioBw/edit ). He referred to https://github.com/apache/mina-sshd/issues/319 which is very detailed.

I have scheduled it for Monday March 6th at 9:00 UTC which is after the UTC morning backport window.

Looks like the images can still be replaced by new version but the thumb being shown is stalled. Sounds like a cache issue of some sort?

For future reference, the configuration change in Gerrit is under refs/meta/config and is commit be3f7e7ac29a305b3ee72261296d1fd15bc756ef (gitiles).