In T216055#5068116, @Nuria wrote:

@mpopov are there any of these metrics we want to remove in the light of the classification that @TJones did on spreadsheet?

As I thought, this is a customization that was added to the English Language Analysis years ago before my time. It was originally limited to search on MediaWiki.org in 2013, and then expanded to all English-language wikis in 2014, but it was never expanded beyond that.

I'll take a look today. I'm pretty sure I know what's happening, but will double check.

Thanks, @Smalyshev & @EBernhardson, for the vendor patch!

After refactoring the lowercase-to-ICU-normalization upgrade code for Greek (T203117) so that the lowercase filter is kept if it is language-specific, I needed to test it for the other language-specific cases: Turkish and Irish. The impact is positive but small because it is limited to the plain field and other fields besides the text field (where the lang-specific lowercasing is already in effect because the analyzers have not been unpacked). Full details on MediaWiki.

Unpacking the Greek analyzer exposes the lowercase filter, which is upgraded to icu_normalizer, losing the Greek-specific processing therein! So, we need to keep the Greek lowercasing even if we do ICU normalization. After that, everything is copacetic. Full write up on MediaWiki.

We need to reindex, but not until after the ES6 upgrade is complete, and LTR has been disabled.

@EBernhardson, thanks for the explanation!

The spikes on create_index are pretty extreme, with 194s for chi-eqiad-with-archive and 291s for omega-eqiad-with-archive. Is that just bad luck, or is something going on with the archives that makes this sometimes take much longer?

In T63080#650462, @Manybubbles wrote:

Bleh. It looks like that symbol is turned into a text boundary by the standard analyzer which isn't nice.

Cool! Thanks, @Smalyshev!

Woo hoo!

In T215966#4952468, @gerritbot wrote:

Change 490412 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] admin: reset Julia SSH key

https://gerrit.wikimedia.org/r/490412

Removing this from current work and moving it to the "Language Stuff" backlog. I'm the only one who could work on this this quarter, and I'm a bit out of my depth with the integration. We'll reprioritize this for future work when we can assign a slightly larger team (≥2 people) to work on it.

@Julia.glen, I think this patch should give you an account, but as user juliaglen. You may need to add User juliaglen to your ssh config.

Hmm—what about Nori (the Korean analyzer) and LTR? I believe we have to disable LTR for Korean, enable Nori, gather more data, then rebuild the LTR model. Sounds like maybe all of that should wait until after the ES upgrade, even though it means re-indexing Korean wikis at a later date.

Looks good, and all the detail is much appreciated.

Sounds good to me! If it turns out that the smallest volume languages have trouble, we can fall back to larger languages on the list.

Should be done now—so try again, please!

@Julia.glen, my hue username has the same weird capitalization as Gerrit (Tjones), which I don't use elsewhere.

In T212885#4944841, @Julia.glen wrote:

I am unable to access hue.wikimedia.org with my LDAP account. Could you take a look? Thanks.

What languages should we initially investigate?

Thanks, @Mooeypoo! That looks like it could work. I really appreciate your explanations and your patches!

Everything looks good now. Serbian (et al.) and Esperanto are working as expected. Thanks, @dcausse!

I can log in now. Thanks!

First draft done. Full details on MediaWiki.

By "LDAP" I assume you mean the login for Wikitech, etc. OIT uses "LDAP" to refer to your Google Apps login, too.

Thanks for looking into this, @Mooeypoo! It's too bad that there isn't a way to make it work now, but I'm glad this provides another use case for potential enhancements to OOUI. If the extra functionality ever gets implemented, ping me if you remember!

In T215346#4929559, @Jdforrester-WMF wrote:

Well, users can do var searchInputWidget = OO.ui.infuse($('#searchText')); to get a handle on the OOUI widget. Is that not sufficient?

Thanks, @Dzahn!

In T170099#4920016, @TTO wrote:

I regret not expressing my gratitude or commenting here at the time.

Seems to be fixed now.

Seems to be fixed now.

Closing this because after looking into it a while back I decided that internal confidence isn't really a thing for TextCat to do, and easy things to improve the quality of TextCat results were done.

I think everything here is fixed. ö, ä, and å are all treated as independent letters and using a instead of ä is the same as using u instead of ä, and other diacritics like á are ignored. Depending on whether you use the completion suggester, go feature, or full text search, you get additional suggestions depending on the place of the typos or the frequency of the incorrect word—all as expected.