(In reply to comment #53)

You can use things like {{subst:DATE}}. But then the {{cite web}} template
itself where you include it needs to be substed as well.

You mean something like <ref>{{subst:cite web | … | accessdate = {{subst:DATE}} }}</ref>? That doesn't work.

Try using {{#tag:ref|{{subst:cite web | … | accessdate = {{subst:DATE}} }} }}

That's great, but subst'ing {{Cite web}} (etc.) would be a disaster. Simply for "Subst:" this is a basic problem which people have had to code round in Javascript and bot code. Leaving this bug unassigned means increasing amounts of resource are used to code for these special cases (which, of course rely on the underlying templates (where appropriate) not changing) as more tools are developed to work on Wikitext.

For example SmackBot (now Helpful Pixe Bot) used to use "Subst:CURRENTMONTHNAME" to date tags. Mainly because of this bug I had to write code to regenerate the XML for AWB to use the explicit monthname, a side effect being that all changes to the XML had to be back-ported to the component model. Moreover if I didn't regenerate the XML the wrong month was used.

Now AWB is no longer used for this I use the current system ides of the GMT month, which is pretty good, but still creates an unneeded dependency on my system clock.

Similarly there are probably workarounds for the other extensions which are not processed but the idea of having a mark-up language that works one way on parts of the page and another on other parts, except through design, is a real bad one.

Although it doesn't currently cause _me_ problems -

I'd like to see this prioritized.

• Bug 29775 has been marked as a duplicate of this bug. ***

*** Bug 30135 has been marked as a duplicate of this bug. ***

Can I give another example of what I think is the same issue, the inability to perform the pipe trick within {{#ref:}}

Example that does not work (at enWS)

{{subst:#tag:ref|[[Author:Charles Darwin|]]}}

should expand to give <ref>[[Author:Charles Darwin|Charles Darwin]]</ref> though it just replicates <ref>[[Author:Charles Darwin|]]</ref>. I am not seeing other issues, though I am willing to undertake trials if there are tests that are desire.

(In reply to comment #59)

Can I give another example of what I think is the same issue, the inability to
perform the pipe trick within {{#ref:}}

Example that does not work (at enWS)

{{subst:#tag:ref|[[Author:Charles Darwin|]]}}

That will never work— subst only applies to templates and #tag is a magic word. What you are trying is equivalent to <subst:ref> which will not work.

What will never work? The subst: expansion of the tag? Seems to work perfectly for me and has been for a while.

My issue is that the piping trick[1] stops working when it is inside either the <ref> or {{#tag:ref|...}}

[1] http://en.wikipedia.org/wiki/Help:Pipe_trick

I just noticed that subst isn't working inside of <ref> tags:
https://pt.wikipedia.org/w/index.php?diff=prev&oldid=28035465

I believe this is the same bug.

Pipe trick doesn't work with Translate extension popups, I guess it's this bug again. https://translatewiki.net/w/i.php?title=MediaWiki:Bw-desc-freecol/it&diff=prev&oldid=3550365

(In reply to comment #63)

Pipe trick doesn't work with Translate extension popups, I guess it's this bug
again.
https://translatewiki.net/w/i.php?title=MediaWiki:Bw-desc-freecol/it&diff=prev&oldid=3550365

That does not sounds like this, because it is not wrapping within a extension tag. Edit over GUI works for that title, this sounds like a problem with the edit api. You should fill a new bug for that.

Nx.devnull wrote:

Patch to allow extensions to specify a function that is executed during preSaveTransform

This patch introduces a mechanism similar to the tag hooks, the only difference is that the callback function gets called when the output type is set to OT_WIKI (which normally means during preSaveTransform), and that the enclosing tags with the parameters are not removed.

An extension can register a callback and then do pre-save transformation on its content - this is the best solution in my opinion, since only the extension knows which parts of the content are supposed to be transformed, e.g. in the case of the gallery tag only the description part.

This is a work in progress/proof of concept, in particular the galleryPST function calls parser->preSaveTransform, which might cause problems if it breaks the parser state. This will probably need something re-entrant like recursiveTagParse.

All parser tests passed.

Attached:

Thanks for this patch! We've been in a code slush (http://lists.wikimedia.org/pipermail/wikitech-l/2012-January/057313.html) for a
few weeks so we're just getting around to looking at these.

We're also doing a lot of parser work, so I'm not sure how relevant this is,
but I'll ask them to take a look, but I'll still get someone to look.

wicke wrote:

(In reply to comment #65)

An extension can register a callback and then do pre-save transformation on its
content - this is the best solution in my opinion, since only the extension
knows which parts of the content are supposed to be transformed, e.g. in the
case of the gallery tag only the description part.

I am not too enthusiastic about encouraging extensions to do their own arbitrary pre-save transform processing. In the longer run, general support for extensions working with parsed (and pre-save transformed) wikitext / HTML DOM input could be cleaner and should offer better opportunities for integration with a visual editor. I am working on something like that in the Parsoid parser (http://mediawiki.org/wiki/Parsoid). There is also a simple and undocumented method for something vaguely like this in the PHP parser (the setTransparentTagHook method).

However, I do not have a good solution for partly-parsed and -transformed extensions like gallery. Will give this some thought, and ask others about their ideas.

• Bug 34999 has been marked as a duplicate of this bug. ***

(In reply to comment #67)

I am not too enthusiastic about encouraging extensions to do their own
arbitrary pre-save transform processing.

lowering priority

(In reply to comment #69)

(In reply to comment #67)
> I am not too enthusiastic about encouraging extensions to do their own
> arbitrary pre-save transform processing.

lowering priority

I am not sure how to equate the "lowering the priority" for a significant issue that doesn't follow normal wiki behaviour? We get numerous responses akin to "I don't like it" however no solutions.

Cite must be one of the most commonly used extensions in wikipedia space, and one would think that it is used multiple times per page. We have a fairly common practice with the renowned pipe trick, and now at comment #69, you are saying that we are "lowering the priority". How about we consider the basic users and solve some of the base underlying issues that bug users every day, or isn't that sexy enough to gather attention? For some of us this is a large annoyance, where we have to create work arounds, by scripting buttons, or doing significant copy and paste.

wicke wrote:

@Billinghurst: Cite can be covered by something like bluehairedlawyer's patch (attachment 5952 / comment 31), as its entire content is parsed as wikitext.

• Bug 35954 has been marked as a duplicate of this bug. ***

lambdav wrote:

More than 6 years after this bug as been reported, the bug is still there !
How many time should we wait for correction ?
Priority must not be Low !

lambdav wrote:

Priority = High because blocking another High priority bug.

Nx.devnull wrote:

(In reply to comment #67)

(In reply to comment #65)

> An extension can register a callback and then do pre-save transformation on its
> content - this is the best solution in my opinion, since only the extension
> knows which parts of the content are supposed to be transformed, e.g. in the
> case of the gallery tag only the description part.

I am not too enthusiastic about encouraging extensions to do their own
arbitrary pre-save transform processing. In the longer run, general support for
extensions working with parsed (and pre-save transformed) wikitext / HTML DOM
input could be cleaner and should offer better opportunities for integration
with a visual editor. I am working on something like that in the Parsoid parser
(http://mediawiki.org/wiki/Parsoid). There is also a simple and undocumented
method for something vaguely like this in the PHP parser (the
setTransparentTagHook method).

However, I do not have a good solution for partly-parsed and -transformed
extensions like gallery. Will give this some thought, and ask others about
their ideas.

If the problem is that extensions could do arbitrary processing instead of the standard PST, then the only solution I can think of is to allow extension to somehow specify which part of their input should be transformed, and then let the parser do the processing. Implementing that would probably be much harder and more cumbersome.

The only thing that needs to be done on my patch is that I'm not sure the extension calling parser->preSaveTransform is 100% safe. I can give it a go if you want, but I don't want to waste my time if the patch will be rejected because my approach (letting the extension do the PST) is not the correct one.

New parsers are nice and all, but they do not fix the problem that we have right now, and they probably never will if you want to keep the current gallery syntax.

bluehairedlawyer wrote:

Actually David's wrong. This bug is seven not six years old. If its not fixed soon we'll have to figure out who's going to pay for it to go to college...

What would be wrong with running the all the contents of a <gallery> through preSaveTransform? I realise it's not wikitext but then preSaveTransform isn't the full parser? What particular pre-save transforms would corrupt the gallery syntax?

Nx.devnull wrote:

(In reply to comment #76)

Actually David's wrong. This bug is seven not six years old. If its not fixed
soon we'll have to figure out who's going to pay for it to go to college...

What would be wrong with running the all the contents of a <gallery> through
preSaveTransform? I realise it's not wikitext but then preSaveTransform isn't
the full parser? What particular pre-save transforms would corrupt the gallery
syntax?

I don't know off the top of my head what characters are allowed in filenames (in particular, is [, { or ~ allowed), but in theory:

<gallery>
File:Oh noez [[This is not a link.png|]] yeah, I know, this is a contrived example
</gallery>

There's also replacing tildes, substing and the reverse pipe trick. Replacing tildes would be the most realistic problem (if ~ is allowed in filenames)

[ { are both disallowed in any page name (which includes images). Three or more ~'s in a row is also forbidden in page titles. So you don't have to worry about that.

Personally I favour an approach of adding an interface for adding parserHooks that allows specifying options (such as if the parser hook should allow PST).

Actually David's wrong. This bug is seven not six years old. If its not fixed
soon we'll have to figure out who's going to pay for it to go to college...

That's still a good 11 years away. We're cross that bridge when we come to it.

bluehairedlawyer wrote:

Some parser tests

Attached:

bluehairedlawyer wrote:

Allow extensions to specify tag contents can be passed through preSaveTransform without being escaped

This is an alternative to Nikola's patch. It allows extensions and internal setHook functions to specify that there contents can be run through the preSaveTransform function without being escaped.

It passes all the current parser tests and a few others I've managed to throw at it.

Attached:

sumanah wrote:

bluehairedlawyer, could you put this patch in Gerrit?

bluehairedlawyer wrote:

Dan Collins has already done this (which is just as well as I haven't figured out how to do it yet). Here's the url:

https://gerrit.wikimedia.org/r/#/c/8784/

That was a smart way to solve it.

Commented some concerns on gerrit.

patch moved to gerrit, removing patch need review.

Dan Collins has two patches, not one:
https://gerrit.wikimedia.org/r/#/c/8784/ (previously mentioned)
and
https://gerrit.wikimedia.org/r/#/c/8785/

• Bug 42078 has been marked as a duplicate of this bug. ***

Somebdy should really give those patches some love, as apparently the original committer has disappeared.

Gabriel: As you assigned this report to yourself a year ago, do you plean to rework the two patches?

Gabriel: As you assigned this report to yourself two years ago, do you plan to rework the two patches? Or are they "somehow" obsoleted in the meantime?

Change 8784 abandoned by Siebrand:
(Bug 2700) Pre-save transform inside extension hooks

Reason:
I'm abandoning this patch set as it's been open for a long time without any outlook at the open issues being resolved.

If the submitter or anyone else would like to work on it again, this patch set can be re-activated by clicking "Restore Change". Please only do this if you are actually going to work on it immediately.

https://gerrit.wikimedia.org/r/8784

• Bug 72079 has been marked as a duplicate of this bug. ***

Change 8785 had a related patch set uploaded (by Paladox):
(bug T4700) Pre-save transform inside extension hooks

https://gerrit.wikimedia.org/r/8785

Change 8786 had a related patch set uploaded (by MarkTraceur):
Pre-save transform inside extension hooks

https://gerrit.wikimedia.org/r/8786

I was tooling around with this, but...

https://www.mediawiki.org/wiki/Manual:Tag_extensions#How_do_I_render_wikitext_in_my_extension.3F

This section says that tag extensions will never have PSTs applied to them. So maybe this bug is WONTFIX and we should abandon all patches for it?

This bug is not only/mainly about tag extensions: see pipe trick, subst, signatures.

Change 272916 had a related patch set uploaded (by Cenarium):
Pre-save transform inside extension tags

https://gerrit.wikimedia.org/r/272916

The bug is referenced at dewiki Wikipedia:Technik/Werkstatt#insource