Dependency Extraction Webpack Plugin impaired DevX #35630
Comments
L-2 policyL-2/L-n policy means that the plugin should support only the latest It's to be implemented on the plugin level only. I think it's a nice policy to start with, but we should not stop here, as we still face all the DevX frustration even with |
Build/automate WP, WC, plugin version mapsI think that all the data is there (somewhere) we just need a nice and automated way to gather it all in a single place. As mentioned in the comment by @jsnajdr
I’d only add to make it machine and human-readable, and for WooCommerce as well. Conceptually it’s gathering all Thanks to that a developer, support engineer, merchant, further tools could inspect what versions are used where and decide upon that.
It solves 3, 5, 6; helps with 2, 4. I think it's something we should start with, as it would give us an insight and data to reason about the problem. |
Make Dependency Extraction Plugin handle dependency versionsI believe we have all the data in place to solve the problem, assuming we already have a way to get granular dependencies’ versions for a given WordPress version. The local I’m not very experienced with Webpack. But to me, the feature that DEWP brings is similar to what native Import maps do: “Check what dependencies are already available in WP/WC and map those imports to external modules” (instead of adding and looking them up in the local bundle). What’s cool about native import maps (besides the fact, it’s native, already available for free in Chromium, and does not require us to complicate our tools and stack) is that they seem to solve the problem of multiple versions problem – with scopes. So if someday we’ll switch to native ESM, we could use a single map for all plugins, without a need for each plugin adding DEWP to their tool stack. I’ll use import maps syntax as I believe it’s clear and declarative enough to express the behavior we’d like to have: Consider the given WooCommerce version uses "@woocommerce/components": "5.1.2",
"@woocommerce/currency": "3.1.0",
"@woocommerce/number": "1.2.3", Then the import map for a plugin that would like to use shared dependencies would look like "imports": {
"@woocommerce/components": "/wp-content/…/woocommerce-admin/…/components.js",
"@woocommerce/currency": "/wp-content/…/woocommerce-admin/…/currency.js",
"@woocommerce/number": "/wp-content/…/woocommerce-admin/…/number.js",
}, That’s what we have today. And AFAIK, that’s where DEWP functionality ends in terms of versions. But hopefully, we can add a bit of checking logic there. OverlapIf our plugin uses "@woocommerce/components": "^5.1.1",
"@woocommerce/number": ">=1.1.1 <1.2.0",
"fast-json-patch": "^3.0.0", Then all the shared dependencies are there, so the bundle will include only Newer versionIf our plugin uses "@woocommerce/components": "^5.1.1",
"@woocommerce/number": "^2.0.0",
"fast-json-patch": "^3.0.0",
"imports": {
"@woocommerce/components": "/wp-content/…/woocommerce-admin/…/components.js",
"@woocommerce/currency": "/wp-content/…/woocommerce-admin/…/currency.js",
"@woocommerce/number": "/wp-content/…/woocommerce-admin/…/number.js",
},
"scopes": {
"/wp-content/plugins/gla/": {
"@woocommerce/number": "/wp-content/…/gla/…/number.js",
},
}, (So the imports originating from Native import maps are resolved on the run-time, so they could use the WP/WC dependency maps from the currently running setup. It solves 1, 2, 4, 5, 7, 8. |
Add runtime import map shim/resolverThis one could be the trickiest to implement, but if a given WordPress version knows its own granular dependency versions and receive the plugin's map, then theoretically it knows whether the versions match. If so it would do what it does today, and return the import from the WP bundle. If not could point back to the plugins bundle. So the "extended" bundle would be requested only if the currently running WP environment does not have a matching dependency. However, I don't know enough on how WP handles all the scripts, to propose something more precise, or judge how doable it is. It may solve all the problems, but would require a lot of changes across the stack. |
Just spitballing some ideas: Sounds like Webpack Module Federation could help here. The official documentation says
If we replace Maybe we can replace DEWP with Webpack DLLs. Maybe we can publish NPM packages with the DLLs for specific WP versions (eg: This package could also provide a set of So MyPlugin depends on Although that would mean I need to maintain several "lines" of MyPlugin: one compatible with |
I also have one thing that I think causes a lot of pain points when it comes to working with the DEWP. And that is the inability to tree shake. If you use a singular function from Regarding your comments, I actually like the idea of a runtime import map shim/resolver quite a lot. Core currently doesn't ship with multiple versions of the packages and I'm not sure whether it would be wise to do so. Of course, you could always bundle all your dependencies with your plugin but as you mentioned that leads to the same code being imported by multiple plugins and therefore a lot of overhead. So the idea of only loading a bundle if the version isn't a match is very intriguing. I also have no clue however how feasible it would actually be to implement something like it. |
Forgive me for the generic title, but I'd like to tackle many interconnected issues related to Dependency Extraction Webpack Plugin (DEWP) and its DevX, as I believe we need a holistic view to solve those.
TL;DR
Plugin developers have no control or even introspection over the extracted dependencies. Not only the versions but also the packages being extracted. Which heavily decreases the quality of our products and the maintenance cost.
I'm trying to gather here the problems and ideas. So, we could refer to this issue from GH issues and PRs with potential solutions (like #35106 (comment)), without losing the overview of the problem.
Context
I'm a WooCommerce(WC) plugin developer, so the perspective below could be WC-oriented. However, I believe the problems stated here are not unique to WooCommerce or my plugin and apply to any other WordPress plugin. WooCommerce adds just another layer of dependencies.
Dependency Extraction Webpack Plugin (DEWP)
AFAIK, the main goal and reason why we use DEWP are to avoid delivering duplicated JS packages from plugins to a single WordPress (WP) site. It reduces the network and CPU cost for users and saves us from a number of problems related to the packages which have a side effect, which may collide when imported/used twice.
I see dependency extraction as a nice way to address that, also I believe that customers' costs and experience should be the top priority when looking for a solution.
But the way our tool works introduces a number of struggles for developers. IMHO, the impact is severve. I noticed a number of people working on different plugins and products discussing their struggles with DEWP. To somewhat evaluate the impact in the plugin I work for, I marked all issues that are affected by this problem (However, it doesn't include everyday frustration)
Where we are at?
Let’s draw a picture of what dependencies and problems we face while developing a plugin.
Naturally, we have (let’s call them “main”) dependencies: WordPress and WooCommerce. We have a strict requirement – driven by the actual merchant/customer expectation – to support at least a few versions behind, not only the latest, as updating the stack is a cost and burden for our merchants. However, given the finite resources we have, we’d rather balance it not to support too many combinations with legacy code. We have a defined process to track updates of those, and a tool (
wc-compat-checker
) to support the PHP side of it. But it's still manual labor, and we do not have much tooling to support the person doing a compatibility check on the JS side.We have “granular” dependencies – individual npm packages stated in
package.json
. There are alsocomposer
dependencies, but I’m not sure if they are also that problematic. I guess, there is no customer-driven requirement for those. I doubt we have customer requests like “Hey, I’d like AutomateWoo to work with my@woocommerce/number@1.9.0
-based store”. However, we do have some constraint that comes from using many dependencies – they need to be cross-compatible with each other and with the WordPress & WooCommerce being used.Then we have dev dependencies, for which the merchant should not care about at all. Unfortunately, they are also tied to WP/WC versions being in use.
Problems
The above creates three fundamental problems:
@woocommerce/components
version being used for respective WC version.To solve/mitigate the first problem, there are
@wordpress/
and@woocommerce/dependency-extraction-webpack-plugin
. However, the way it works makes the whole development quite indeterministic: which packages are extracted, which main or granular dependency is actually used in runtime? It’s being extracted and used blindly, regardless of versions, without even reporting what was extracted, and what is assured to have a specific version. That created a bunch of other problems:@woocommerce/components
. Unless, I carefully, manually, and deeply curated and extracted all the granular dependency trees across all (minor and patch) versions of main dependencies. Then all I get is still a range of versions.This is very time-consuming labor, that needs to be done very often. When adding a dependency, when updating anything, when debugging an issue, when checking compatibility. Currently, there is no tool to support that or even a specific list of resources to track.
That may result in unexpected behavior and error. Checking that again requires even more manual effort. As it requires digging into the source code of DEWP at a given version and manually comparing packages list.
That makes reproducing the reported problems harder and more time-consuming, eventually affecting customer experience.
package.json
while the one run by the customer is totally different.In a Slack discussion, @nerrad suggested implementing the L-2 policy, and supporting only two versions behind, then supporting the lowest versions available. This naturally limits the ranges and number of combinations but does not tackle the problems themselves. Plus, may fail if any main dependency decreases the version of its dependency, or introduce backward-incompatible change.
Cost of status quo
In my opinion, the above impair not only DevX but also innovation and the quality of our products.
Innovation
Quality
Ways to go
I think when looking for a solution for the problems stated above, we could take a few (non excluding) strategies:
Personally, I'd start from the latter, as it's the cheapest to start with (in time, effort, and chaos it'd generate).
//cc all folks who I noticed discussing related ideas: @ecgan, @roo2, @scinos, @jsnajdr, @fullofcaffeine, @noahtallen, @sirreal, @gziolo, @mcsf, @nerrad
Solutions
I don’t have any precise well-established solution for the above. I’d rather start a discussion about those. But here are few ideas I managed to gather or come to my mind. I put them in separate comments, for easier reference.
The text was updated successfully, but these errors were encountered: