Planet Plone - Where Developers And Integrators Write

Questions for the December Steering Circle?

Posted by PLONE.ORG on December 06, 2021 12:05 PM

As described in the Foundation's July 2020 discussion of Plone governance, a series of Steering Circle meetings is being held to discuss our organizational structure and processes, and any hot topics of the moment. This is part of the Foundation's initiative to solicit ideas for changes that will better serve the needs of our community, our projects, and our teams. The meetings will be held every two months, and the next one will be December 14th at 13:00 UTC (15:00 CEST). Each Plone team will send one or two representatives, including the Zope, Volto, RestAPI and Guillotina teams.

The Steering Circle meetings will include a discussion of questions from community members. Please use this form to submit any questions you have and we will put them on the agenda.

Thank you for being an awesome community and for helping to move Plone forward into its third decade!

Plone Conference 2021 Recap

Posted by PLONE.ORG on November 27, 2021 08:15 PM

Plone Conference 2021 Online

This year's conference was a 9-day event for the Plone, Zope, Volto, Guillotina & Pyramid communities and consisted of training, talks, and sprints. The conference provided insight into the long history (20 years!) of Plone CMS as well as the latest, future-proof features and visions.

Plone - The original, open-source, enterprise-grade, all-in-one content management system written in Python.
Zope - The original Python web application server - the foundation for Plone and inspiration for Guillotina.
Volto - Plone 6's snappy, modern React front end powered by the RestAPI.
Guillotina - A re-imagined asynchronous back end compatible with Plone's RestAPI.
Pyramid - A small, fast, down-to-earth Python web framework.

https://2021.ploneconf.org/

With 295 attendees from 30+ countries from over the world, from Jamaica to Japan, USA to the UK, Germany to Finland, to Austria, Brasil and Australia, the conference attracted many newcomers to the community, in addition to long-time contributors since the early days of Plone.

With the help of our sponsors, like Six Feet Up, iMio, and many others, the conference was held 100% Online, using LoudSwarm platform by Six Feet Up. LoudSwarm nicely combined video streaming, recordings, schedules, and chat discussion.

Every training, talk, and presentation was recorded and will be available online on Plone YouTube channel later.

Participants

Group Photo of Plone Conference 2021 Online.

Training

As in previous years, the conference included over 30 hours of free training, with topics including Mastering Plone, Volto Addons, Pyramid, and Guillotina.

https://2021.ploneconf.org/schedule/training

The training videos are already publicly available on YouTube.

Talks and Open Spaces

The 4 days of talks included almost 60 professional presentations, ranging from addressing specialized use cases to building and managing projects and into very technical talks about some aspect of a certain technology. Many of the talks focused on Volto, Plone 6's new React-based frontend, and there was also a lot of discussion about the future of Plone 6.

With Open Spaces and 5 min lightning talks, it was possible to bring into focus many other aspects and topics.

Some highlights of talks:

Plone community
Plone 6, Volto
Plone 6 Classic
Plone.org renewal
Volto Add ons
Plone Migrations
Abfab, Frontend development
Quanta - new style for Plone
Plone in Brazil
7 things to surprise you with Plone development
Guillotina
Accessibility

And many more! https://2021.ploneconf.org/schedule/talks

Keynote speakers

Sprints

Every Plone Conference includes at least two days of sprinting, so after the conference talks, people continued to meet, working to improve and develop the Plone ecosystem.

Sprint topics included e.g.

Plone 6 roadmap
Volto
Documentation
Image handling
Docker support for installers
Plone.org renewal

Community

The Plone community aims to be the most welcoming and friendly community towards new users and veterans alike. We are happy to report that this conference brought many new faces and we sincerely hope that everyone felt the love.

Chrissy Wainwright addressed some of these communal aspects in her first keynote: https://2021.ploneconf.org/schedule/state-of-the-plone-community

Every conference has a party, and this was no different, except for the fact that everyone was online this time.

Thank you, everyone, sponsors, organizers, trainers, presenters, and especially participants for making this year's conference one to remember!

Next year: Plone Conference 2022 in Namur, Belgium!

Next year the conference will be held in Namur, Belgium!

So mark your calendars and stay tuned for Plone Conference 2022 news! Follow @ploneconf and #ploneconf2022 on Twitter and Instagram.

Plone Foundation Board for 2021-2022

Posted by PLONE.ORG on November 24, 2021 04:42 AM

The Plone Foundation exists to further the development, marketing, and legal affairs of Plone and the Plone community.

Among many tasks, the Plone Foundation board:

supports development of the Plone ecosystem by coordinating team efforts and identifying vital sprint targets;
protects the integrity of the Plone ecosystem by monitoring rights and assuring the continued value of the ecosystem intellectual property;
coordinates marketing of the Plone ecosystem and provides funding for representation at conferences and related events;
coordinates World Plone Day and an Annual Conference to encourage community connections and the enjoyment of the world's best CMS.

Read more about the governance process at https://plone.org/foundation/governance-process

Board members for the next year are:

Érico Andrei (President)
Jens Klein (Vice President)
Andy Leeb (Secretary)
Paul Roeland
Víctor Fernández de Alba
William Fennie
Kim Paulissen (new board member)

You can email the entire board at board@plone.org

David Ichim: What's new in Plone 6 frontend for developers

Posted by Maurits van Rees on November 08, 2021 11:18 AM

Distilled from the latest work done in Volto, we're showcasing some patterns, features, or enhancements that have landed in Volto from the last year to the present. We will also have a glimpse of what is ahead in the future of Volto with the new features roadmap.

In the past year we had four major releases, 40 minor releases, 36 alpha releases, 25 patch releases, for a total of 105 releases. Plus some new tooling and tool releases, like plone i18n and plone generator.

New Volto config, dubbed as Volto's Configuration Registry, introduced to fix circular import dependency problems. Hot Module Reloader was fixed. Read the upgrade guide.

New i18n (internationalisation) infrastructure. This is now a separate package. Same is used for generation of i18n in add-ons.Read the upgrade guide.

Forms as schema. Forms should be constructed from schemas instead of by hand. InlineForm allows us to create forms for blocks from a schema. Blocks can have variations, or we can extend it. Read the edit component documentation.

New widgets:

Object List Widget. Similar to the original DataGridField. Used in core by the Search Block facets.
Object Browser Widget is now a separate widget, instead of part of a block, and now allows the addition of external content.
Querystring Widget. Behaves like its counterpart from plone.app.querystring. Allows to create search criteria, used by the search block.
URL Widget. Used on text inputs, it knows to validate their value as a url, both internal and external.
Vocabulary Terms Widget for a JSONField, acts as a source for a SimpleVocabulary or Choice field. Play with it in the storybook.

New core blocks:

Search block
Teaser block (ongoing work).

Pluggable framework, similar to React's Portal component. See the talk by Tiberiu and see the Pluggables development recipe.

Storybook provides a sandbox to build and test visual components in isolation. Currently it is only setup to be used by Volto core. We need help with work to have storybook setup with adding. See the Storybook talk held by Victor, and see the storybook itself.

Critical CSS: inline the critical CSS for improved performance. Run critical-cli to output critical.css. This is then inlined in the headers, while regular CSS is moved to the bottom of the body. See the deployment documentation.

Lazy Loading utilities. Introduced injectLazyLibs HOC wrapper to inject lazy-loaded libraries as properties to your components. These components are only loaded once all your main libs are loaded.

Express.js middleware. Volto uses this for SSR and static resources serving. You can now write custom middleware and add it to settings.expressMiddleware.

API expanders allow the expansion of different API endpoints from Volto with calls from your custom endpoints. Avoid adding too many expanders if they are not critical to the initial page.

External routes: useful when another application is published under the same top domain as Volto. If Volto would fetch the content for that path, it will break. You can disable this path in config.settings.externalRoutes. You can also use regular expressions.

Seamless mode, introduced in Volto 13, enhanced in Volto 14, which has already seen a lot of alpha releases. Originally, we tried to unify both frontend and backend servers under the same path, but this was tricky, causing various problems. We settled on a new ++api++ traversal for getting information from the backend. Also, to come closer to zero config, you can now pass environment variables at runtime instead of build time. This means you can generate one build, and use this in all environments (testing, production). Read the deployment documentation.

Context navigation component. This is a navigation portlet component, similar to Classic Plone. The view is there, but you need to enable it. See the development recipe.

There is some work in progress:

Slots are Volto's answer to portlets, see the Volto Slots talk by Tiberiu Ichim for more details.
Image proxy: image scale generation done by a middleware instead of plone.scale.
Authentication from backend.
Replace Draft.js editor with Volto Slate. What is missing is a migration tool from one to the other. But work has started on a block conversion tool.
Async blocks that work with SSR.
Defaults in blocks form.

Future work:

Defaults in all widgets
Enable blocks enhancers in all blocks
Storybook in add-ons
Use newest react-intl package
Refactor the folder contents component
Form editing text enhancements, making it easier to modify text inputs.
A "group block" included with Volto
Quanta toolbar

I did nothing, I just brag about what others have done. So thank you Volto early adopter community!

Tiberiu Ichim: Volto Pluggables

Posted by Maurits van Rees on November 08, 2021 09:55 AM

This presentation is an introduction to the new Volto developer-targeted feature, the Pluggables framework. It is more an argument for extensibility in CMS UI and in Volto.

Basically, with Pluggable a central component provides a pluggable slot that other components can fill, like this:

<div className="toolbar">
  <Pluggable name="toolbar-main" />
</div>
// ...
<Plug id="quicklinks" pluggable="toobar-main">
  <Button />
</Plug>

This is a Volto port of https://github.com/robik/react-view-slot.

The big picture:

I work with Eaudeweb Romania
Our client EEA is an early adopter of Volto
The strategy is: move everything to Volto
But the EEA sites are less brochure, the CMS side is really strong.
We build powerful UIs for power users.
The EEA already has 91 Volto repositories on GitHub. How can we scale that? Can we write an add-on to make it easier to write an add-on?

In React, data flow is top to bottom. A parent component passes properties to children and children communicate with the parent by emitting events. This makes sense and works well. For "out of tree" data you need Redux. There is no protocol for add-hoc communication between components.

UI state is fluid. Extensibility means reusability and scalability. This is hard. You need to design upfront. Plone backend uses the Zope Component Architecture, which means pluggability is baked in, it is very natural. You can view Pluggables as viewlets-on-demand, but they are really not. But yes, you can think about a Pluggable as a viewletmanager and a Plug as a viewlet.

You can overwrite a Plug with a Plug, by registering it with the same id. So if the original Plug gives you color blue, you can overwrite it with color red.
You can do custom rendering of Plugs within your Pluggable, by iterating over all Plugs and for example wrapping each in a div with a class name.

Showcase: volto-workflow-progress (main toolbar plugin) and volto-editing-progress ("sidebar" for plugin).

Limitations:

No Server Side Rendering
Watch out for dependency lists.
Limited adoption for now.

Implementation in pseudocode:

First the context:

<PluggablesProvider>
  <Pluggable name="top" />
  <Plug name="delete" />
</PluggablesProvider>

Then the dumb version of the Plug:

const Plug = ({id, children}) => {
  const { register } = useContext(PluggablesProvider.Context);

  React.useEffect(() => {
    register(id, () => children);
  });

  return null;
}

Takes a bit of study, but in the end it is not so hard.
The Pluggable becomes a bit simpler:

const Pluggable = (name) => {
  const { getPlugs } => useContext(PluggablesProvider.Context);
  return getPlugs(name).map(f => f());
}

Site updated from Plone 2.5 to Plone 6.0

Posted by Maurits van Rees on November 04, 2021 11:34 AM

I am one of the Plone Release Managers, and have been working on Plone 6, which is now in alpha stage. But my personal website was still using the ancient Plone 2.5:

Screenshot maurits.vanrees.org Plone 2.5

Often I have made plans to update my site to:

Plone 3
Grok
Plone 4
Plone 5
Plone 5.2
finally Plone 6

Long ago it was clear to me that an inline migration would not be practical. It would take too many steps: update the code to Plone 3.3, migrate the data, to Plone 4.3, migrate data, to Plone 5.2 Python 2.7, migrate data, Plone 5.2 Python 3, migrate data, Plone 6, migrate data.

Additionally, the question was how to handle the weblog, which is the main content. This was using Products.Quills, a Plone add-on for blogs. Updating to Plone 3 could have worked at the time, but this was made harder by some some custom code I added. This enabled me to use it as a podcast. I used this to enrich some of my summaries of sermons from my church with the actual audio of the sermon. I doubted whether to even include this content in Plone 6, as the last sermon was from 2008. I hate breaking links, so I kept it, although a bit hidden.

Another point that needed some extra attention, was that most, if not all, blog entries were not written in html, but in ReStructuredText. I make a lot of summaries of talks when there is a Plone Conference. The html editor on Plone 2.5 did not work anymore, or I disabled it to a simple textarea. I always open up the same text editor that I use for programming (previously Emacs, currently VSCode), and type the summary there. I much prefer writing ReStructuredText there, especially when I simply need text without any markup. I then paste it in Plone, without fear of losing all my work when my internet connection dies.

Lastly, I have an RSS/atom feed which is used by planet.plone.org and maybe others to stay updated when I add a new blog entry. I did not want this url to change.

Anyway, about six years ago I decided that I would use collective.jsonify to export my site, and then import it using transmogrifier. But time passed without any progress. This year, collective.exportimport was shaping up to be the preferred way to import the data. For export you can use it in Plone 4.3 as well, but definitely not in Plone 2.5.

At the beginning of this week I looked at jsonify. Didn't I have a local copy of my website on my laptop, with collective.jsonify installed? No! And it was not installed on the live site either. And installation would be hard, because the site uses Python 2.4, and currently you cannot even reach PyPI with older versions of Python 2.7.

Mildly panicked, I started on a custom script to export the content, still as json. Some details aside, this actually was not so hard. I looked at what collective.exportimport expected, and created the Python list of dictionaries accordingly. Then do a simple json.dumps() call and you are done. Except that this gave an ImportError: the json module is not available in Python 2.4. Okay, install simplejson package. But you need PyPI access for that, which does not work. Workaround:

Manually download an egg of Python 2.4-compatible simplejson 1.7 and save it in the buildout directory.
cp bin/instance bin/instance-json
Edit the new script and add the simplejson egg to the system path.
bin/instance-json run export_mvrsite25.py

After that, it was not too hard anymore. I used plonecli to create a new add-on with Plone 6.0.0a1. I actually do not yet use the add-on code, except for loading a minor patch that I added, but this gave a reasonable, modern Plone 6 buildout. Create a Classic Plone site, tweak a few settings (let Folder use folder_workflow, allow English and Dutch, navigation depth 1, enable syndication, configure caching and mail), import the content with collective.exportimport, edit a bit, and done.

The weblog now consists of standard Folders and Pages. To improve the view, I added a Collection, showing the latest pages first, and showing all content of the last seven blogs. I enabled syndication here, so it has an RSS/atom feed.

The weblog has always advertised two atom feeds:

One for all weblog entries, at https://maurits.vanrees.org/weblog/atom.xml
One for weblog entries with keyword 'plone' at https://maurits.vanrees.org/weblog/topics/plone/@@atom.xml

In the new site, the first one kind-of worked out of the box, but it only showed the items that were directly in the weblog folder, and this is not where my weblog entries are. I solved this with a patch to Products.CMFPlone.browser.syndication.views.FeedView: when atom.xml is viewed on a folder, check if it has a default page, and show the atom.xml of this default page instead. In this case, the default page is a Collection with the correct settings. So the general feed will keep working.

For the second one, my first idea was to create a folder 'topics' and within it a Collection with id 'plone'. Problem: 'plone' is a reserved word and cannot be used as id of a content item, so the id became 'plone-1'. Solution here: create the 'plone-1' collection directly in the weblog folder, and do a redirect in the frontend server (nginx):

 rewrite ^/weblog/topics/plone/@@atom.xml /weblog/plone-1/atom.xml last;

And that's it! My website is now on Plone 6.0.0a1:

Screenshot maurits.vanrees.org Plone 6.0

There are some more details that I could go into, like splitting up the buildout into multiple parts, with tox as main way to build everything, in preparation for moving more and more parts to pip-only. But that will have to be for another time.

Meanwhile: have a look around, and enjoy the fresh look!

Steering Circle

Posted by Maurits van Rees on October 31, 2021 02:39 PM

Volto 14 alpha 23 is out. So still in alpha, but companies are using it in production. Should be final soon. Some plans for Volto 15. Created plone.volto integration package, where we try to give an easy transition from earlier company-specific versions. plone.restapi as always is pretty boring, stable. Erico worked on Docker integration.

Plone 6 alpha 1 is out. Eric sent an email for some coordination, like docs, training, accessibility, installers. If you want to be involved, let me know.

Franco has stepped out of the Framework Team, thank you for all your work. There is discussion about the role of the Framework Team. Plan is to keep it running, some more people have been asked.

Membership team: we have some people in sight as new members. Erico is stepping down as team lead, Victor is stepping up.

Security: Plone 6 will have 5 years security support. Synchronizing with Zope team. Some new members may be coming.

Marketing: busy with conference, also after the conference. Busy with plone.org renewal.

Installers: see the talk by Jens Klein earlier. Plone 6: no more installer, but we do have tooling. There are Docker images. We may want to reduce the role of buildout, and focus more on pip.

Plone Conference: you are looking at it. Some tasks to do afterwards. If anyone is interested in getting a certificate for following a training, ask us, and we can send it.

Internationalization: new branches for Plone 6, so Plone 5 uses different branch. New releases for 5 and 6. Updating po files, looking at i18n in Mockup.

Admin/Intrastructure: servers are still running. Cat herding sysadmins for doing stuff, keeping things up to date.

Trainings: relaunch is complete. We have three new trainings: Theming Plone Classic, Deploying Plone 6, Getting Started with Plone 6 (that one only in video). Various have seen major updates. Need to work on landing pages (we have two), remove the number 5 from the url, update some more trainings. Maybe Mastering Plone Classic training, but hard with navigation due to duplicate section targets when copying. Migration training would be good. We need to prune and tag some.

Plone Classic: We did polishing on Barcelonate, it is pretty ready. Focussing on bobtemplates before the trainings, making theming easier. JavaScript/ES6 is the remaining big issue. Plan is to finish it this year, we are quite far. We need other people helping us out.

Documentation: will be releasing a new Plone 5 branch today. For the new stuff, the tech stack is ready. New version of automated screen shot is about ready. We don't want a duplicate of the training, but we can automatically include code of it, so there is only one source of truth. The style guide is not always followed, seeing what we can do about that. Biggest point is missing documentation. There are now branches where the various teams can add and edit their content. We may change things, but we take this as input for the final structure.

Fred van Dijk: 7 things that can surprise you when you start customising or developing for Plone

Posted by Maurits van Rees on October 29, 2021 09:06 AM

This is a basic rundown and summary of our beloved subjects like ZODB persistence. Traversal. The view/viewlet/portlet trinity. How is a call handled in Plone. The differences between zcml and generic setup. Utilities and the ZCA. restrictedTraverse. The Plone catalog.

These are surprises that I have encountered myself, or that I have seen on faces of people I have trained or worked with during the years.

Everything can or could be done through the web (TTW).

Zope vision from the nineties. Why don't we use this dynamic language called Python so we can change things TTW? It's so easy.

The learning curve.

It starts easy, but then you hit what we call a Z-shaped learning curve. Dynamic Python makes things easy, and then it makes things hard, at least when you put Zope on it, then CMF, then Plone. Plone the product on top of a CMS on top of a framework on top of a language. We have a product, a framework, a stack, so it is hard.

Five levels of conceptual complexity.

It helps to teach all the levels. Give new users a drawer for each level, so they have a box that they can put some info in.

You have:

browser: html, css, js
frontend: transforms, templates, zpt, metal, diazo
application logic: views, viewlets, portlets, adapters, Zope Component Architecture
dynamism: GenericSetup, zcml, xml, zope schema
programming language: Python, buildout, pip
package WorkManager (OS): apt, rpm

4 Same language/formats on different levels:

XML is used for the ZCA, GS, zope.schema, Diazo rules
package manager: buildout, pip, setuptools, GS

But: there is no magic. It is just advanced technology.

Startup:

Python uses sys.path modules to start bin/instance
ZCA loads site.zcml, package includes, other zcml to change configuration.
Then we have a runtime environment with objects and classes, the ZODB. GenericSetup is then some XML that you can use to change the ZODB.

So the ZCA overrides components in runtime. The alternative is to edit core files, maybe compile them, and restart. Much less nice and not sustainable.

So now we have a Plone process running.

Zope is not that complicated.

Over HTTP Zope gets a request, does traversal, finds an object in the ZODB, loads it in memory. Then on top of this object we show a browser view / template. The template hooks into a main template, maybe does some sub requests, some Diazo, and we end up with some html and we are done.

This is all still 'lies to children'. It is simplified until we are able to understand more. With increment exposure to these concepts, it will stick more. It is complicated, but there is no magic.

Acquisition.

It is traversal in the wrong direction.

When you try to explain things, you improve your own understanding.

There is so much Plone content online: training, docs, youtube, github, discourse. We all learn in different ways, with own preferences, reading, viewing. There are so many repositories on github that you can explore for new ideas. Just yesterday Philip did a talk about exportimport and afterwards I did it, but from a different angle. It helps.

The community is our biggest asset.

Annette Lewis and Will Gwin: From Zope to Plone: Thinking User-First During Migration

Posted by Maurits van Rees on October 28, 2021 02:39 PM

Migrating a site is always a challenging task, but when you have dozens of subsites with specific brand standards and custom user functionality, the challenge becomes mammoth. Six Feet Up worked hand-in-hand with Purdue's College of Engineering to migrate their existing Zope site and its subsites into a new Plone instance running on Plone 5.2 / Python 3. Throughout the migration process, we considered the project scale, timelines, and limiting the impact on end users, all while managing the balance between user needs and best practices. During this presentation, you will learn:

why it matters to think user-first during migration,
about creative solutions for translating content and functionality into Plone, and
how to successfully migrate subsites.

An overview of the project:

HigherEducation always seems to go a bit slower, certainly with migrations.
Our previous CMS was built in Zope and was getting extremely old.
We have been using Python since 2001.
There were security concerns and modernization issues.

The impact:

Only 15+ content editors.
40+ public facing subsites
30,000 total pages.
20+ departments and units

Why did we select Plone as the next CMS?

It is a modern CMS solution
Python-based, so that fit what we currently have.
It is built on top of the Zope web framework that we were already using.
We looked at Drupal, Wordpress, and more, but that would have been a too big undertaking.

This was in 2018.

Laying out the solution. From requirements to action. Who are our users and what do they want? Not just our direct client (Will and his team) but their clients/users.

Challenges during development:

Purdue University changed its brand look. We had to seamlessly blend subsites into the existing parent site.
Convert all content types and templates from Zope to Plone.
Keep sight of the user experience in both environments. Could they use the new environment without too much training, or needing to have too much tech knowledge?

Determine the project essence. Distill the requirements down into broad categories: accessibility, usability, flexibility, security.

What is the path to successful collaboration?

The absolute best might not be the right answer.
It's okay to say no to an idea, but you dhouls have an alternative ready.
Aim for the best, avoid the dangerous, end up somewhere in the middle sometimes.

On to our development goals:

Do things in a Plone way. Plone uses Zope, but Zope may do some things in a different way than is the best way in Plone.
So observe best practices.
Make it intuitive and keep it familiar for the editors.

Solutions at a glance

Migrating site content:

We wanted to move subsites one by one, as needed.
Translate existin content to Plone content types
We could re-import content over existing content non-destructively.

Theming: retrofitting Plone into an existing theme. That is what Diazo was made for, bridging the gap between Plone and the theme, especially since the theme is 'living', with subtle changes coming in often.

Each subsite had a browser view named local.css to change some things. Not really what you want in Plone, but they really needed it, as a way to make subsites or sections look different. So we added an action to edit the local css, inheriting from parent folders.

We created a subsite settings control panel. We used lineage.registry for this. All kinds of customizations can be done based on that, for example add extra text and links in the navigation menu. They used to be able to do this in Zope as well, but that was with various properties, and much more code oriented.

We use Mosaic for flexibility of layout. In Zope we had blocks for layout. Mosaic took this a step further into a nice UI with drag and drop. It gives faster site prototyping and development.

The sandbox: we wanted to have safe spaces for content experimentation. Completely separate from production environment. It is used for new user training and testing. It is quick and easy to reset.

This migration project has been a constant collaboration between the Purdue communications office, Engineering Computer Network, and Plone company Six Feet Up. The content editors feel empowered to make complex changes, without constant oversight from my team.

Alan Runyan: Building a Secure Cross Platform Mobile/Tablet Application (Flutter) using Plone as Backend Server

Posted by Maurits van Rees on October 28, 2021 01:44 PM

Enfold has been working on a secure cross platform mobile application the past eight months. Walk through of the Requirements, Security, Flutter framework, Backend configuration of Plone, Authentication and Lessons Learned. Our goal is to have a free public release of a limited version of the application Q4 2021.

It was an adventure for us. The core team never built a large mobile application. We did not know what we did not know.

The big picture:

A mix of devices (Android, iOS) needs to synchronize files to and from Plone.
All services are self-hosted in GovCloud. So we have no central database or server that we control.
If this becomes a success, then future phases may require a lot of certification of the codebase, code reviews.
Initially we worked with 15k devices, supporting 40k would be a success, the ceiling that we might support is 300k.
Users are completely offline for longer times.

We used Flutter to create a React native app, see https://flutter.dev It is a UI toolkit. Why did we use Flutter?

It is cross platform mobile.
It uses Dart, which is statically typed, making code analysus much easier.
Google seems to be prioritizing developer user experience, it really shows of quite a bit.

Dart has asynchronous code as a first class citizen. Quite different from Python. Runtime reflection (pdb) is unavailable. It has good ergonomics, with generics and closures. It is a driving force behind the Flutter toolkit.

Thoughts on mobile development:

It is a lot to take in.
You need to understand lots of languages, for us: kotlin/java, swift/obj-c, and Dart.
No idea how to test platform integration.
Native libraries are managed using Cocoapods/Gradle. Flutter drives those. Setting it up is yet another new thing to learn.
There are lots of inconveniences, like how do you read sqlite off a device, because that is how we store some of the info?
Also inconsistencies: if the app works on an emulator, that does not mean it works on a device.

On the server side:

Plone operating as a Webdav server
We need to support OIDC (mod_openids/oauth2)
Not many writes, maybe 100-1000 per day, but lots of reads.
20k+ devices daily

Alternatives to the server/protocol could be nice:

Honestly, are there any standards other than WebDav?
An S3 api would be reasonable.
So ideas are welcome, let me know.
We have been working on prototypes with guillotina_webdav + openidc.

The good parts of Flutter:

UI/UX development is very fast, with lots of widgets. We had two developers who were used to Angular, and they took it up quite fast.
The bridge to native code (Pigeon) is straight forward.
Drift is an amazing sqlite library.
Riverpod for state management
Dependency management is good (flutter pub). You can tell that they learned a ton from others, like pip. Except that very occasionally the package cache is broken so you need to clean it.
They have a good community, with add-ons.

The not so good parts of Flutter:

Inconsistent platform features, like WorkManager (Android) versus NSUrlSession (iOS)
Dependency churn: often new versions come in, which you then need to check.

The mobile app:

We are still wrapping up the remote file operations.
After we deploy into production, we will improve the UI.

Yes, we hope to open source the synchronization framework, and maybe the foreground/background transferring subsystem. Yes, we have built a Flutter web-app of this, but it looks just like the mobile app currently. Needs a separate layout really. No, we have not done a Desktop app.

Lightning talks Wednesday

Posted by Maurits van Rees on October 27, 2021 08:35 PM

Peter Holzer: My first Plone 6 site

This was in April 2020. I friend wanted a shop for his post cards. Lots of search filters.

We create a new Plone theme on Bootstrap 5. Updated our bda.shop stack.

See https://www.fotoeigenart.ch

Michael McFadden: Web developer confessions

I had a web form up where people could submit confessions, totally anonymous.

I used Perl
I sometimes test things in production.
I voted against removing the tag from HTML.
I store user passwords, as we often need to login as a user to fix their... stuff. We take precautions, but still.
I allowed editors to add css at a top level in the site.
I have a marquee tag on our site.
I am using iframes to include those who are unwilling to move into our big Plone Site. It is a way to do imports.
I use pdb on production all the time with a client instance.
I use for my page layout.
I don't setup the whole translation machinery on our Plone Site. Hello USA!
I use heavily duplicated code. What's wrong with coding with copy and paste.
I did not backup my data, and I knew the RAID aray was bad.
I think downtime is good for mental health.
I still hate javascript. And now we have node.
I don't always do web development. But when I do, I do it through Zope using plone_custom.css on a production site.
I fixed it with css. .class {display: none !important}
I suggested to a user to try it in another browser, like Edge. I am still ashamed.
I am not trained in UX or web design, but I do it anyway, because I just know what's right.

Fulvio Casali: Plone sponsorships

We are looking for financial sponsorship on a regular basis. Open source is a labor of love, but we also need money. The Plone Foundation supports the community financially for sprints, conferences, releases. We have administrative costs, like for trademarks, lawyers, hosting.

So you support independent media producers via Patreon, Substack, Medium, Youtube? How much? If not, maybe your boss does? Have you watched listened to the Plone podcasts?

Sponsors can be providers, customers, universities, companies that somehow use Plone. We couldn't do it without you.

Mail sponsor@plone.org.

Dylan Jay: Why Drupal won down under

Two stories of government websites, in this case UK and Australia, where other CMSes won the day. Maybe with Plone 6 we can gain some ground back.

PreviousNext is a Drupal shop in Australia. In 2012 aGov launched, a distribution of Drupal specifically for Australian Government websites. That was cleverly done. They organized a Drupal conf in Canberra, the capitol. Most of the stuff could not have been done without support by Acquia, the behemoth company behind Drupal. It helps to be able to say that they have got you backed.

Then the UK. In 2012 Gov.uk launched. 2013 Service manual published.

Ideas for Plone:

Open CMS for a specific government.
Acquia equivalent
How to sell o governments that have been burnt by Plone?
Target internal React developers.
Use a better term than 'enterprise'. Governance CMS for React Developers?
Concentrate on services, not websites?

Maik Derstappen: Current state of Mockup

We are moving Mockup and Patternslib to ES6. We replaced some Mockup patterns with patterns from Patternslib. Most mockup patterns are finished, just a few left that need a bit of work. Plone Classic UI frontend works basically, some issues left in control panels mostly.

Things will get easier. No more RequireJS yelling at you. Add-ons can provide, require and ship any javascript module. Plone will only load everything once, thanks to Webpack module federation.

Timeline: ES6 branch will be merged in the coming months.

Give it a try using the buildout.coredev repository and:

./bin/buildout -c plips/plip-3211-mockup-redone.cfg

Help us! Join us during the conference sprint or every Thursday on Discord, the classic-ui voice channel.

Johannes Raggam: Javascript integration

This is about Webpack module federation.

Goals:

add-on without recompiling
No or negligable code duplication, small bundle size

Hard to achieve with Plone 5 and RequireJS.

New concept: Webpack module federation, or actually just module federation, as it is not tied to webpack.

separate bundles
dependencies between each other
define shared bundles
define exported and remote modules
fallback to own dependency

David Glick: Snowfakery

Snowfakery is a tool for generating fake data that has relations between tables. Every row is faked data, but also unique and random, like a snowflake. To tell Snowfakery what data you want to generate, you need to write a Recipe file in YAML.

You may want realistic data in tests, but not production data.

I write a yaml file and tell it how many folders and documents to create. I let snowfakery export this to json. Actually, I have defined a slightly different format for Plone. This outputs a json file that can be used by collective.exportimport.

Snowfakery documentation: https://snowfakery.readthedocs.io/en/docs/

Philip Bauer: One small VS Code trick

For the training this year for the first time I used one editor for both Volto and Classic. This is because of one VS Code trick.

I use the Pylance language server for VS Code, which is fine, but it cannot initially find the Plone code. Simple fix for that: I have a script zopepy in my bin folder. Find it, copy the path, in VS code configuration go to 'Python: Select interpreter', and paste the path. After a few seconds, or maybe reloading an open file, it works, and VS Code finds all packages and modules.

Erico - New Docker images

Plone loves containers. Some have had a love/hate relationship with Docker, but we are over it. Maybe Docker is over it too.

We have a new generation of Plone Docker images:

- ``plone/plone-backend`` 5.2.6 and 6.0.0a1
- ``plone/plone-frontend`` Volto 14.0.0-alpha
- ``plone/plone-zeo`` is still available (5.2)
- ``plone/plone-haproxy``

Use your own Plone image, with an example extra add-on:

FROM plone/plone-backend:6.0.0a1
RUN ./bin/pip install "pas.plugins.authomatic --use-deprecated legacy-resolver"

The use-deprecated option should hopefully not be needed in the near future. Maurits has opened a [PR in pip](https://github.com/pypa/pip/pull/10574) for that.

All these images are based on pip. One point: do not use autoIncludeDependencies in your zcml, as this does not work with pip.

Examples for Docker compose: https://github.com/plone/plone-frontend#using-docker-compose

Thank you to the installers team: Jens, Alin, Silvio, Steve, Kim, Erico.

Thomas Schorr: Pyruvate WSGI Server Status Update

Posted by Maurits van Rees on October 27, 2021 07:23 PM

At last year's Plone conference, I presented Pyruvate, a WSGI server implemented in Rust (and Python). Since then, Pyruvate has served as the production WSGI server in a couple of projects. In this talk I will give a project status update and show how to use Pyruvate with Zope/Plone and other Python web applications and frameworks. I will also present some use cases along with benchmark results and performance comparisons.

WSGI is the Python webserver gateway interface. It is the default way for Python apps to talk with a webserver.

During the Python 3 migration of Zope and Plone, WSGI replaced ZServer as the default, since Plone 4. The ZODB is not thread-safe. This means there is a limited choice of WSGI servers. The Zope docs recommend only two: waitress and Bjoern. Other popular servers showed poor performance for Zope.

Rust is a new, popular programming language. It can also be used to extend Python packages. The cryptography package does this, and we all use this package.

Pyruvate from a user perspective:

pip install pyruvate

Then create an importable Python module:

import pyruvate

def application(environ, start_response):
    """WSGI application"""
    ...

pyruvate.serve(application, "0.0.0.0:7878", 3)

Using pyruvate with Zope/Plone. In buildout you add pyruvate to the eggs of your instance, and use a different wsgi-ini-template.

Pyruvate supports active Python versions, currently 3.6-3.10.
Using Mio to implement I/O event loop.
Worker pool based on threadpool, 1:1 threading.
A PasteDeploy entry point.
It is stable since 1.1.0.
Supports Linux and MacOS
Hosted on Gitlab
There are binary packages for Linux, so there you don't need a Rust compiler to install it.

Let's see about performance. As starting point: Performance analysis of WSGI servers by Omed Habib, see Appdynamics blog. This is a performance comparison of 6 WSGI servers, published in 2016. Tested were: Bjoern, CherryPy, Gunicorn, Meinheld, mod_wsgi and uWSGI.

Benchmarking tool was wrk. The test WSGI application simply returns some headers, and the text "OK".

I changed the original setup. Swapped Meinheld and mod_wsgi for Pyruvate and Waitress, swapped CherryPy for Cheroot. Use Python 3 only. Now we look at the metrics.

Number of requests served:

Bjoern is by far the best.
Pyruvate does not do so well with lower number of simultaneous connections, but with higher numbers, it jumps to second place after Bjoern.

CPU usage:

Bjoern is single threaded, so is 100 percent, the others can do more.
Gunicorn starts the best, but drops in performance with more simultaneous connections.
Pyruvate starts slightly below it, but sustains it better.

Memory usage:

More or less okay for all.
Except uWSGI, where memory usage steadily goes up.

Errors:

With increasing load, all servers show errors on higher load.
Except waitress and Pyruvate.
uWGI always shows errors for every single request, so something is wrong. But it still serves the request. So maybe an issue of the benchmarking tool.

Why is Bjoern so much faster?

Good implementation, many optimizations.
It is single threaded. Switching from single to multi-threaded comes with benefits and costs.
Shared access to resources adds to complexity.
Offloading requests to worker threads only makes sense when there is something to work on, which is not the case in this benchmark.
Python's GIL generally makes multithreading less effective.

Let's look at benchmarking Plone 5.2.5. Testing with Bjoern, Pyruvate and Waitress. All three serve the Zope root about 600 requests per second, and this stays the same when simultaneous connections increase. Serving the Plone root: all three about 27 requests per second.

Number of errors, mostly socket errors:

Bjoern starts giving errors a bit earlier, starting from ten simultaneous connections, but it holds up well.
Pyruvate and waitress start giving errors at 50 simultaneous connections.
With 100, Pyruvate does a lot worse than Bjoern, and waitress even more than that.
So you really can't have that many simultaneous requests to Plone.

CPU usage is the same, all 100 percent. In this setup I have used one worker for each server.

Serving /Plone as json: Bjoern is slightly better than Pyruvate, which is slightly better than waitress, but all are around 500 requests per second, quite close to each other.

The most interesting is number of requests served when getting a 5.2 MB blob from blobstorage. Bjoern is around 200, Pyruvate 180, so close, and waitress a lot worse, dropping from about 80 to 40.

Next setup: 2 threads for Pyruvate and waitress. Pyruvate is then better than Bjoern. Waitress starts better, but cannot keep up.

Using 2 threads but 4 CPU, Both Pyruvate and waitress are a bit better than Bjoern, and keep it up.

Conclusions from benchmarking:

Bjoern is the winner when using a single worker for all URLs except /Plone.
When serving a more complex page such as /Plone, there is no real difference in the number of requests served, but Bjoern is showing errors a bit earlier.
Adding one thread plus sufficient resources lets both Pyruvate and waitress perform better than Bjoern.
All configurations failed to sustain higher loads, more than 50 connections.
Bjoern and Pyruvate are serving blobs a lot faster than waitress.
Pyruvate can challende waitress in all scenarios.
When adding worker threads, Pyruvate seems to make better use of resources than waitress.

As test I setup Apache for fair balancing between two ZEO clients on Plone 5.2, one served by Pyruvate, one by waitress. Consistently more requests are sent to Pyruvate (53 percent).

Code: https://gitlab.com/tschorr/pyruvate

Tiberiu Ichim: Volto slots, portlets on steroids

Posted by Maurits van Rees on October 27, 2021 03:55 PM

Problem: Volto has no equivalent of a viewlet. Solution: slots. They can be management slots, presentation slots, below-footer slots.

One reason: we try not to customize the main template.

Volto also does not have portlets. Well, if you really want them badly enough, you can have them. There is a PR in plone.restapi to export portlets, so you could render them in Volto.

Idea: reuse Volto blocks for layout.

Plone has had portlets for a long time, and it is very useful, especially for smaller sites. You should not have to be a web developer to change the website layout. Portlets give site administrators some power to influence the look of their own site. We should keep that possibility.

Volto's slot proposition:

Simplify configuration. Portlets in Classic need too many files.
Volto blocks are very expressive.
Require Modify Portal Content permission for slots.
UI Power: give more capabilities: - atomic blocking of parent blocks - override parent blocks

How can we use them?

Sidebars: listings, info boxes, navigation
section headers, content

Current status: big PRs on plone.restapi and Volto. Overall the basic functionality is 60-70 percent ready. I will do a live demo.

Questions for the October Steering Circle?

Posted by PLONE.ORG on October 25, 2021 06:35 PM

As described in the Foundation's July 2020 discussion of Plone governance, a series of Steering Circle meetings is being held to discuss our organizational structure and processes, and any hot topics of the moment. This is part of the Foundation's initiative to solicit ideas for changes that will better serve the needs of our community, our projects, and our teams. The meetings will be held every two months, and the next one will be October 29th at 13:00 UTC (15:00 CEST). Each Plone team will send one or two representatives, including the Zope, Volto, RestAPI and Guillotina teams.

The Steering Circle meetings will include a discussion of questions from community members. Please use this form to submit any questions you have and we will put them on the agenda.

Thank you for being an awesome community and for helping to move Plone forward into its third decade!

Plone 6.0.0a1 Alpha Is Here!

Posted by PLONE.ORG on October 22, 2021 07:50 PM

Plone community proudly presents the first alpha release of Plone 6 - feel free to try it out!

https://plone.org/download/releases/6.0.0a1

Some Highlights of the Alpha Release

The big one: Volto as new front-end, using React, built with modern JavaScript tools.
The backend is now called Plone Classic. It generally works the same as Plone 5.2, so if you are not ready for Volto yet, you can just use this.
Support only for Python 3.7, 3.8 and 3.9.
Zope 5.3
Extensive overhaul of Plone UI elements based on Bootstrap 5 components.
Barceloneta LTS theme
Add control panel for relations
Add plone.api.relation module to ease using relations.
Use Dexterity for the Plone Site root object.

Plone 6 editing experience combines the robust usability of Plone with a blazingly fast JavaScript frontend

Plone 6 editor

Plone 6 Uses the Latest Web Technologies

The Plone 6 user experience has been redesigned from scratch to be awesome for power users and occasional users alike. It's easy to create adaptive layouts using a flexible and powerful blocks system built from the latest web technologies. There's no need for in-depth knowledge of how the web works or how the page will look on thousands of different devices - editors can split individual blocks into multi-column content that will automatically adapt to any device. Building complex, responsive pages with Plone 6 is effortless.

The blocks system is not limited to rich text. It will come with a faceted search block that allows editors to easily create sophisticated searches that make use of content metadata. Other blocks can present collections of content dynamically generated from a search. Plone has always included search, now the blocks system allows the search to be harnessed in new ways.

Form Builder, No-Code Content Types, Rich Ecosystem

Plone 6 also includes:

A powerful form builder. Editors can create forms via drag and drop and define actions such as sending form data via email or storing it in CSV format for later use;
No-code custom content types that can be created through the web, including metadata fields, behaviors and view templates;
A rich ecosystem with more than 100 add-ons, even before it has been launched; an active and enthusiastic community of developers and companies is creating new add-ons every day.

The quick growth of the Plone 6 community is the result of developer-friendly technology choices. It is built with the two most popular programming languages (Python and JavaScript) and an open-source software stack that powers some of the largest websites in the world.

Read more about Plone 6.

Try Plone 6 and learn more at the Plone Conference 2021 Online

Try Plone 6 at https://plone.org/what-is-plone/plone/try-plone
Install the Plone 6 Alpha at https://plone.org/download/releases/6.0.0a1
Learn more about Plone 6 at Plone Conference 2021 Online starting October 23rd - Get your tickets at https://2021.ploneconf.org/

Plone Conference 2021 - Schedule, Trainings and Participation Info

Posted by PLONE.ORG on October 19, 2021 05:05 PM

Taking place over 9 days, from 25th to 31st October 2021, the Plone Conference 2021 Online will feature training, keynotes, talks, open spaces, sprints, and social activities.

Schedule

Important dates:

Training will happen over the weekend - October 23 & 24
4 days of keynotes and talks + 1 day of open spaces - October 25 - 29
Sprint - October 30 & 31

All information can be found at: https://2021.ploneconf.org

Check the full schedule at Schedule with precise starting times (CET).

Training

Two days of training is included in the ticket price. Learn about topics like:

Mastering Plone 6
Volto Add-On Develoment
React and Volto
Plone 6 Classic UI Theming
Volto and Plone Deployments
Getting started with your Plone 6 site
Guillotina React
And more!

All training info can be found at https://2021.ploneconf.org/schedule/training

All trainers are friendly professionals from the Plone community.

Important! Remember to register to training at using registration form.
(Just make sure you have a ticket first).

Keynotes

We will have three amazing keynotes:

State of the Plone Community by Chrissy Wainwright, Plone Foundation President
Plone 6 Power and Control by Timo Stollenwerk, founder of kitconcept and Volto evangelist
From Opaque to Open: Untangling Apparel Supply Chains with Open Data by Katie Shaw, Chief Programme Officer, Open Apparel Registry

Check the schedule at https://2021.ploneconf.org/schedule

Talks and speakers

There will be over 40 professional talks by speakers from all over the world. Topics will range from Plone 6 and Volto to theming, migrations, case studies, documentation, and much more. Talks about Guillotina, React, Python, Accessibility, Open Source, Community, UX, Frontend and Backend development are scheduled.

Talk audience level is clearly described, so there will be something for everyone through 3 tracks. All talks will be streamed live and will be available as recordings immediately after. Participants will be able to discuss with the speakers after every presentation.

https://2021.ploneconf.org/schedule/talks

https://2021.ploneconf.org/speakers

The conference will also feature one day of lightning talks and open spaces for bringing up ideas and innovation!

Get your tickets

Only 100$ per ticket, and only 25$ for people from developing countries

https://2021.ploneconf.org/tickets

How to participate

This year’s conference will be entirely online, through the LoudSwarm virtual platform, the same as with the 2020 conference.

Each registered person will get an email about how to access the system. Rest assured it will be very easy, and the level of interaction will be high! This is a community event, with lots of discussion and friendly emojis :)

Join the Slack spaces

The Plone Conference Slack Workspace is where all information about the event will be shared. This is also where you will be able to chat with speakers, attendees, and organizers. If you haven't already, join the Plone Slack now and navigate to channel #conf2021_hallway.

If you have a question during the event, please visit the #conf2021_help_desk channel and our team will be available to assist.

Code of conduct

The Plone Foundation is dedicated to providing a respectful, harassment-free community for everyone.

https://2021.ploneconf.org/code-of-conduct

Follow Plone Conference

Follow Plone Conference https://twitter.com/ploneconf
Follow Plone at https://twitter.com/plone
Contact the organizers at conf@plone.org

https://2021.ploneconf.org/

Contact the organizers at conf@plone.org.

The Plone Newsroom Podcast

Posted by PLONE.ORG on October 18, 2021 12:40 PM

The Plone Newsroom is a monthly podcast brought to you by Philip Bauer and Fred van Dijk. Technical and non-technical topics will be covered, including Plone, the Plone community, and whatever else they come up with to keep us informed!

Suggest a topic to include in the next episode by sending an email to Philip or Fred (first name 'at' plone dot org).

The premier episode featured a discussion of Plone Open Garden and a roundup of conference and release news, plus information about collective explicit acquisition and the Volto Search Block. Go to the Newsroom page to view other episodes.

Plone Store Is Now Open - Get Your Plone Gear!

Posted by PLONE.ORG on October 13, 2021 12:00 AM

The Plone Foundation proudly presents:

https://store.plone.org

The new shop for all Plone-related gear from T-shirts and hoodies to stickers and such is now open!

For now, you can get:

The shop is built on the TeeMill platform by the Plone Marketing Team, and we will be adding more Plone-themed gear over the coming weeks and months.

We made an extra effort to bring the site online before the Plone Conference 2021 online, so if you order your conference T-shirt now you might get it just in time before the conference opens 23rd October!

The Plone Foundation provides all items in the store without any markup, so the prices should be relatively low (Shipped from UK, so take note of customs and shipping fees).

All clothing items are high-quality certified organic products, made from post-consumer remanufactured organic cotton in a renewable energy-powered factory, audited for a wide range of social and sustainability criteria. Read more at https://plone.teemill.com/the-journey.

The Plone Foundation Welcomes Two New Members

Posted by PLONE.ORG on October 12, 2021 12:00 AM

The Plone Foundation welcomes two new members after unanimous confirmation by the Foundation's Board of Directors on September 30, 2021.

Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.

Nicola Zambellos presenting at Plone conference

Nicola Zambello

Nicola has been developing with Plone since 2016 and has become an important contributor to the Volto project. He started RawMaterial, a Plone-based company, last year and will present his vision for a practical Green Web strategy at this year's conference. "I love Plone," he writes. "I feel at home with the community and I strongly want to support and invest in Plone, taking it to new horizons while maintaining what makes it so good."

Nicola will present one of the trainings at this year's conference; he lives in Ferrara, Italy.

Tiberiu Ichim

Tiberiu works with EauDeWeb and has been a member of the Plone community since 2004. Since 2019 he has been focused on Volto, and is a member of the Volto Core Developers team. He is the initial author or major contributor for several Volto add-ons, including volto-slate (alternative rich text editor) and volto-block-style (generic styling for Volto blocks).

Tiberiu lives in Oradea, Romania.

The Plone Foundation encourages applications from long time contributors to the Plone project and community. Learn more about the Foundation membership and the application process.

Nominations Open for Plone Foundation Board of Directors

Posted by PLONE.ORG on September 27, 2021 07:00 AM

If you have an interest in helping the governance of Plone, and particularly the energy and time to pitch in, please consider nominating yourself to serve on the Plone Foundation board of directors for 2021-2022.

Nomination Process

Log in on plone.org and go here:
https://plone.org/foundation/meetings/membership/2021-membership-meeting/nominations
Add a page there with your name in the title.
For the body, discuss:
- Who you are
- Why you're interested
- What you think you can add to the Plone Foundation
- Most importantly, the name(s) of one or more Plone Foundation members who "second" your nomination
Once ready, click "submit" in the workflow drop-down menu to get a reviewer to look at your nomination.
Nominations will be accepted until October 22 2021, 23.59, UTC. The election will be conducted in conjunction with the annual meeting, which will take place during the Plone Conference 2021. All active members of the Plone Foundation will be eligible to vote.

About Board Membership

The Plone Foundation is a not-for-profit, public-benefit corporation with the mission to "promote and protect Plone". That has meant that the board is involved in:

protecting the trademark, copyrights and other intellectual property, including considering licensing and usage issues;
hiring the release manager;
working with sub-communities like Zope, Guillotina, and Volto
working with various committees, including marketing and membership;
handling "other stuff in the community" as needed
but not: directing Plone development. The board facilitates, but does not direct, the development of Plone itself.

While there's lots of work that happens online, much of the critical business of the board is conducted during video meetings every two weeks — typically, board meetings last about an hour to 90 minutes though occasionally they can run over to handle time-critical issues. Please consider whether this fits your schedule, since missing more than an occasional meeting severely limits the ability of the board to reach quorum and conduct business.

Historically, board meetings have been organized to occur during daytime hours in America and evening hours in Europe, currently at Thursday nights, 19.00 UTC in northern hemisphere summer and 20.00 UTC in northern hemisphere winter. That can always change with new board members.

In addition, there is a board mailing list (private), where we discuss things in addition to the meetings.

This is a working board. Be ready to regularly take on and complete responsibilities for board business.

The board writes no code and makes no development decisions. It is much more concerned with marketing, budgets, fundraising, community process and intellectual property considerations.

You do not need to be a Foundation member to serve on the board (in fact, board leadership is an excellent way to become a Foundation member). All you need is to get an active Foundation member to second your nomination.

The Plone Foundation is interested in broadening the diversity of our leadership, with regards to gender, ethnicity, and geography.

If you have questions about the nomination process, contact the board: board@plone.org

Plone.org Is Getting a Facelift

Posted by PLONE.ORG on September 16, 2021 04:21 PM

Years have passed since the 2016 sprint at Penn State where a team of community members worked on a new theme and madly reorganized content on the Plone 5 version of plone.org. The site dates back to 2002 and the Plone 1 days, and the software and content had been upgraded in place over the years with only minor theme changes - to Plone 2 and 2.5, then Plone 3, then Plone 4, and finally Plone 4.3. It served us well, but because Plone 5 brought many changes, including a new out-of-the-box theme (Barceloneta), we mounted a major effort to refresh the design as well as upgrade the content and software.

What was new then is now looking old, and the marketing team has embarked on a modernization effort. The ultimate goal is to upgrade to Plone 6 and create a React-based theme using the new front end. But meanwhile we've been having a series of mini-sprints to improve what we have now.

Our first major initiative was to improve the News section, which holds an amazing collection of content. Browsing it can take you back in time - to the Plone 1.0 RC1 release announcement for example, or Alan's 2002 thoughts on what Plone should be, or approval of the Plone Foundation as a 501(c)(3) organization. It was already possible to browse news items by year, but we thought categorization by topic would also be useful. So we tagged every news item, and now you can browse news items by category. Fulvio Casali chronicled this effort in his 2020 Plone Conference talk Oh the Places We've Been!

A not very attractive display of news items and listings was another issue. So we sketched out a cleaner look, with a standardized lead image aspect ratio and a more useful byline. Then the more technically adventurous members of the marketing team (Norbert, Fulvio, Érico) strapped on helmets and figured out how to make changes to the site's theme. You are looking at our initial improvements, and there's more to come.

Our other major initiative is to move the contents of the plone.com site over to plone.org. Over the years plone.com became very difficult to maintain, so we have discontinued it. (Contact the marketing team if you need to retrieve any plone.com content.) With that in mind, we created a What is Plone? section on plone.org which is oriented towards the plone.com audience. It is also a place for us to describe all the pieces of the Plone ecosystem and how they fit together.

In addition to these bigger jobs we've been making lots of little improvements during our mini-sprints, including fixing bugs old and new as recorded on the plone.org issue tracker.

Would you like to help with this effort?

We'd love to have you!

Join our effort to promote Plone by publishing regular plone.org news items - successes, new developments, controversies, generally telling a broad audience what's happening in the Plone world
Do you have design skills? We don't and we need help with design improvements and eventually a new theme for Plone 6
If you are a theming wizard please help us modernize the site styles - more 2021 and less 2016
Show off Plone's built in search by creating a beautiful search results listing
Help us with our ongoing efforts to fix bugs and curate content
Help us migrate plone.org to Plone 6

Please contact the marketing team to get involved. Anyone with technical, design or content editor skills is welcome.

The Plone Foundation Welcomes Two New Members

Posted by PLONE.ORG on September 09, 2021 10:09 PM

The Plone Foundation welcomes two new members after unanimous confirmation by the Foundation's Board of Directors on September 2nd, 2021.

Cléber J. Santos

With over 15 years of Plone experience, Cléber is an important member of the Brazilian Plone community. He was part of the team that helped organize three editions of the Plone Symposium South America and the Plone Conference in Brasilia.

Cléber lives in São Paulo, Brazil.

Steve Piercy

Contributor of many projects under the Pylons Project, including Pyramid, WebOb, Waitress, Colander, Peppercorn, and WebTest, Steve is also a constant presence in Plone community chats and forums. In recent years, he has collaborated with the Plone Documentation and Training materials teams and attended Plone Conferences as a speaker and a trainer.

In his spare time, Steve volunteers technical support for bicycling and environmental activist organizations. He lives in Eugene, Oregon, USA.

The Plone Foundation encourages applications from long time contributors to the Plone project and community. Learn more about the Foundation membership and the application process.

Podcast "The Plone Newsroom" launched

Posted by Starzel.de on September 02, 2021 10:02 AM

Brought to you by Philip Bauer of https://www.starzel.de and Fred van Dijk of https://zestsoftware.nl

Watch it on Youtube Channel https://www.youtube.com/channel/UCPo8CirtIfvt8GO4rm-dD4g

Plone Conference 2021 Online - Tickets for Sale Now!

Posted by PLONE.ORG on August 19, 2021 07:38 PM

The annual conference is a chance for the Plone community to come together to share new developments, success stories, and the future of the community. Taking place over 8 days, the conference will feature training, keynotes, talks, open spaces, sprints, and social activities.

This year’s conference will be entirely online, through the LoudSwarm virtual platform. No matter where you are, you can participate!

Plone Conference website now online

Behold https://2021.ploneconf.org/

The conference website is, of course, built with Plone.

Tickets are now available

Discounted tickets, available until September 30th, also a special price for developing countries.

Get your tickets here.

Submit your talk proposal

The topics can range from Plone, Zope, Volto, and Guillotina to Python and Pyramid or from fancy JavaScript to cool case studies and beyond.

There are many kinds of talk slots, from 5 min lightning talks to 30 and 45 minutes long, and you can target your talk to different audiences too.

Submit your talk proposal!

Stay tuned for more info

Follow Plone Conference at Twitter https://twitter.com/ploneconf

Follow Plone at https://twitter.com/plone

Questions for the August Steering Circle?

Posted by PLONE.ORG on August 11, 2021 11:55 AM

As described in the Foundation's July 2020 discussion of Plone governance, a series of Steering Circle meetings is being held to discuss our organizational structure and processes, and any hot topics of the moment. This is part of the Foundation's initiative to solicit ideas for changes that will better serve the needs of our community, our projects, and our teams. The meetings will be held every two months, and the next one will be August 17th at 2:00 PM UTC. Each Plone team will send one or two representatives, including the Zope, Volto, RestAPI and Guillotina teams.

The Steering Circle meetings will include a discussion of questions from community members. Please use this form to submit any questions you have and we will put them on the agenda.

Thank you for being an awesome community and for helping to move Plone forward into its third decade!

Plone 5.2.5 Released!

Posted by PLONE.ORG on August 06, 2021 07:10 PM

General notes

Plone 5.2.5 is a bug fix release of Plone 5.2. The release Manager for this version is Maurits van Rees.

Note: this is a fresh release. Installers are not ready yet but will be made available.

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.2.5/versions.cfg.

For the Plone 5.2 upgrade guide, see https://docs.plone.org/manage/upgrading/

See https://plone.org/download/release-schedule for the planned release schedule.

Plone 5.2.5

Plone 5.2.5 is a release of Plone 5.2.

Download Plone 5.2.5

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.2.4/versions.cfg.

Useful links:

Some highlights of this release are:

Security fixes in AccessControl and Products.isurlinportal.
Security fixes from Products.PloneHotfix20210518 taken over in core.
Zope: 4.5.5 to 4.6.3
Products.CMFPlone: Add PLONE52MARKER Python marker.
plone.app.iterate: Add proper support for Dexterity folderish content.
plone.folder: restore webdav support.
plone.registry: Allow plone.schema.JSONField to be stored in registry (dictionary-like).
plone.namedfile: Cache stable image scales strongly.
plone.recipe.zope2instance: customize WSGI, profiling, python-env.
plone.restapi: JSONField, sub blocks, use_site_search_settings.
Lots of bugfixes, especially improving Python 3 compatibility.

For detailed changelog, go to https://plone.org/download/releases/5.2.5

Join Plone Chat, Now at Discord!

Posted by PLONE.ORG on August 05, 2021 03:13 PM

There are many ways to reach out to other Plone developers and users. The two most important platforms are the Community forum at https://community.plone.org/ and our chat platform, which has been moved from Gitter to Discord.

Online Chat at Discord

Discord is now the best way to chat with members of the friendly Plone community. There are various channels to choose from.

Join the Plone Discord.

Please remember that you will be chatting with volunteers. Read more about chat info.

Please do not use chat to ask for support. Support questions should be directed to either the volunteers in the Plone forum or commercial providers.

Guillotina and its contributors

Posted by PLONE.ORG on July 01, 2021 08:18 PM

Guillotina is a modern, asynchronous back end designed for building high-performance, horizontally scaling JavaScript applications.

Who are the Guillotina contributors?

First, let’s introduce our new contributors: Roger Boixader and Joan Antoni. They both work at Iskra which is a well-known company in the Plone community as they have been actively using and supporting Plone for years (for decades actually!).

Roger was kind enough to answer few questions about his involvement in Guillotina:

Q: As a developer, where are you coming from and what do you do? How much Python has been in your career until now? 

Roger: My name is Roger Boixader Güell, I come from Berga, Barcelona, Catalunya and I am 28 years old. I studied computer engineering in Girona, Catalunya. I had never used Python before entering Iskra and for the first year I have mainly used JavaScript. My first steps with Python were with Django and then with Guillotina. 

Q: Why is using Guillotina relevant in your technical context?

Roger: Because it is the main framework that we use in Iskra on the backend side. 

Q: What is the thing you like the most in Guillotina? 

Roger: The simplicity to create an application, with a few lines of code of Python or with a YAML file you can start a project easily .

Q: What do you think should be improved in Guillotina?

Roger: Documentation and transactions. 

Q: Have you been involved in an open source community before?

Roger: No, it's my first time. 

Q: What would you expect from the Plone Community?

Roger: I think the Plone Community can help recruiting more Guillotina contributors, and the more contributors we have, the better Guillotina will get!

As we could expect, Guillotina attracts people who are not necessarily connected to the CMS world, but it is quite interesting to see Guillotina is considered as a valuable alternative to Django by young Python developers.

How is the core project managed?

The Guillotina project belongs to the Plone Foundation, just like the Plone project does, but it is obviously much younger and does not have (yet) all the Plone development and decision-making workflows.

So far, there is a periodic meeting every three weeks which allows us to discuss evolutions, important pull requests, and new use cases. Currently there are six participants in this meeting.

The repository is in the Plone Github organization.

When they join, Guillotina contributors do sign the Plone Contributor Agreement (as does any contributor to any Plone Foundation project), but they are not core contributors immediately (unlike Plone contributors). At the moment, there are three core contributors: Ramon Navarro Bosch, Nathan Van Gheem, and Jordi Massip. Their review and approval is needed to merge any pull request.

Apart from the core repository, Guillotina has a full ecosystem, managed in the Guillotinaweb GitHub organization; the most important elements are:

guillotina_elasticsearch
guillotina_gcloudstorage
guillotina_ldap
guillotina_react
guillotina_s3storage
guillotina_stripe
guillotina_volto

Each of these projects has an official manager (not necessarily a Guillotina core contributor).

Sprints

We organize a sprint in southern Europe every year when there is no global pandemic! We are also often involved in Volto sprints (and these happen usually in northern Europe, but also when there is no global pandemic).

You should join!

Security patch 20210518 version 1.5 released

Posted by PLONE.ORG on July 01, 2021 08:05 PM

This is a routine patch. There is no evidence that the issues fixed here are being used against any sites.

Version 1.5 of the hotfix is available from:

https://plone.org/security/hotfix/20210518 – if you grab the zip from here, please check that the version.txt contains 1.5 and/or that the md5/sha sum matches. You may get an older version from the cache. Try adding ?x=1 to the URL if this happens.
https://pypi.org/project/Products.PloneHotfix20210518/

This version is a recommended upgrade for all users.

Zope users are advised to upgrade to Zope 4.6.1 or 5.2.1. If this is not possible, you can try this new version of the hotfix.

See the original 20210518 hotfix announcement

From the changelog:

1.5 (2021-06-28)

Fixed new XSS vulnerability in folder contents on Plone 5.0 and higher.
Added support for environment variable STRICT_TRAVERSE_CHECK.
- Default value is 0, which means as strict as the code from version 1.4.
- Value 1 is very strict, the same as the stricter code introduced in Zope 5.2.1 and now taken over in Zope 4.6.2. There are known issues in Plone with this, for example in the versions history view.
- Value 2 means: try to be strict, but if this fails we show a warning and return the found object anyway. The idea would be to use this in development or production for a while, to see which code needs a fix.
Fix Remote Code Execution via traversal in expressions via string formatter. This is a variant of two earlier vulnerabilities in this hotfix. This was fixed in Zope 4.6.2, which takes over the already stricter code from Zope 5.2.1.

Note: we don't usually release another version almost six weeks after the original one, and three weeks after the previous version, and including a fix for a vulnerability which was only reported last week. However, this contains a fix for a close variant of one of the original vulnerabilities and needs a fix in the same code, so it seemed easiest for the security team and for Plone users who patch their sites to release a newer version.

Security patch 20210518 version 1.4 released

Posted by PLONE.ORG on June 14, 2021 04:11 PM

This is a routine patch. There is no evidence that the issues fixed here are being used against any sites.

Version 1.4 of the hotfix is available from:

https://plone.org/security/hotfix/20210518 – if you grab the zip from here, please check that the version.txt contains 1.4 and/or that the md5/sha sum matches. You may get an older version from the cache. Try adding ?x=1 to the URL if this happens.
https://pypi.org/project/Products.PloneHotfix20210518/

This version is a recommended upgrade for all users.
Zope users are advised to upgrade to Zope 4.6.1 or 5.2.1. If this is not possible, you can try this new version of the hotfix.

See the original 20210518 hotfix announcement

From the changelog:

1.4 (2021-06-08)

Use safe html transform instead of escape for richtext diff. Otherwise the inline diff is not inline anymore.
(Note: I forgot to add this to the changelog on PyPI/plone.org).
With PLONEHOTFIX20210518_NAMEDFILE_USE_DENYLIST=1 in the OS environment, use a denylist for determining which mimetypes can be displayed inline. By default we use an allowlist with the most used image types, plain text, and PDF. The denylist contains svg, javascript, and html, which have known cross site scripting possibilities.
By popular request, allow showing PDF files inline. Note: browser preference plays a part in what actually happens.
In untrusted path expressions with modules, check that each module is allowed. In the first version of the hotfix we disallowed modules that were available as a 'private' alias, for example random._itertools. But if random.itertools without underscore would have been available, it was still allowed, even though itertools has not been explicitly allowed. (itertools might be fine to allow, it is just an example.)

Security patch 20210518 version 1.4 released

Posted by PLONE.ORG on June 14, 2021 04:11 PM

This is a routine patch. There is no evidence that the issues fixed here are being used against any sites.

Version 1.4 of the hotfix is available from:

https://plone.org/security/hotfix/20210518 – if you grab the zip from here, please check that the version.txt contains 1.4 and/or that the md5/sha sum matches. You may get an older version from the cache. Try adding ?x=1 to the URL if this happens.
https://pypi.org/project/Products.PloneHotfix20210518/

This version is a recommended upgrade for all users.
Zope users are advised to upgrade to Zope 4.6.1 or 5.2.1. If this is not possible, you can try this new version of the hotfix.

See the original 20210518 hotfix announcement

From the changelog:

1.4 (2021-06-08)

Use safe html transform instead of escape for richtext diff. Otherwise the inline diff is not inline anymore.
(Note: I forgot to add this to the changelog on PyPI/plone.org).
With PLONEHOTFIX20210518_NAMEDFILE_USE_DENYLIST=1 in the OS environment, use a denylist for determining which mimetypes can be displayed inline. By default we use an allowlist with the most used image types, plain text, and PDF. The denylist contains svg, javascript, and html, which have known cross site scripting possibilities.
By popular request, allow showing PDF files inline. Note: browser preference plays a part in what actually happens.
In untrusted path expressions with modules, check that each module is allowed. In the first version of the hotfix we disallowed modules that were available as a 'private' alias, for example random._itertools. But if random.itertools without underscore would have been available, it was still allowed, even though itertools has not been explicitly allowed. (itertools might be fine to allow, it is just an example.)

Plone Powers a New Registration Portal

Posted by PLONE.ORG on June 04, 2021 03:55 PM

Challenge

The Open University of the University of Jyväskylä in Finland faced a daunting challenge: With a robust offering of international studies courses and a student audience beyond the usual set of university enrollees, registration needed to be faster, less complicated, and more flexible. For example :

Students from countries other than Finland needed a flexible authentication protocol
Students part-way through the registration process needed to get quickly back to the place where they had stopped
Coursework needed to be quickly and easily browsable, and purchase through university systems needed to be seamless
Course administrators needed a system where course listings could be updated with ease
Every step of the process needed to be loggable so that administrators could identify choke points and eliminate them

Solution

The development team at the University of Jyväskylä drew upon the strong core foundation of Plone to handle the all-essential data storage and management in a secure and accessible framework. Using Plone's modern RestAPI, the team was able to integrate, first, a wicked fast front end for managing data entry and browsing. Next, they used similar APIs to link to university systems for payment, user management, and the learning management system involved in course delivery. The ReactJS-based Volto front end (which will be standard on Plone 6) made dedicated and custom layouts very easy for the developers to create and even easier for administrators to add, edit, and organize.

One last integration using the RestAPI addressed the logging challenge. The team incorporated an open-source business process management tool, BPMN, into the platform to provide a window on every transaction, including how long each registration required and when and where registration ran into trouble.

Continuous improvement of the Plone framework has resulted in a fast, easy-to-use front end that editors love, deeply integrated with Plone's highly secure CMS backend built on Python and Zope. Based on Plone's unbeatable security record, the team knew they could trust the data to be safe. What's more, the framework's granular permission management was harnessed to allow teachers to access some areas, while students only got to see their own dashboard.

Feedback

User feedback has been overwhelmingly positive from both teachers and students at Open University. Problems with the identification and registration process have diminished greatly and, surprisingly, this had the side effect of improving the atmosphere in the courses.

More Information

Visit the English language version of the portal.

Read a technical blog post about the business process management integration.

Security patch released 20210518

Posted by PLONE.ORG on May 18, 2021 10:00 AM

This is a routine patch with our standard 14 day notice period. There is no evidence that the issues fixed here are being used against any sites.

CVE numbers: CVE numbers have been assigned; see the individual pages.

Versions Affected: All supported Plone versions (4.3.20 and any earlier 4.3.x version, 5.2.4 and any earlier 5.x version).

Versions Not Affected: None.

Zope: Zope is also affected. See details below.

Nature of vulnerabilities:

The patch will address several security issues:

Remote Code Execution via traversal in expressions. Reported by David Miller.
Writing arbitrary files via docutils and Python Script. Reported by Calum Hutton.
Various information disclosures: mostly installation logs. Reported by Calum Hutton.
Stored XSS from file upload (svg, html). Reported separately by Emir Cüneyt Akkutlu and Tino Kautschke.
Reflected XSS in various spots. Reported by Calum Hutton.
XSS vulnerability in CMFDiffTool. Reported by Igor Margitich.
Stored XSS from user fullname. Reported by Tino Kautschke.
Blind SSRF via feedparser accessing an internal URL. Reported by Subodh Kumar Shree.
Server Side Request Forgery via event ical URL. Reported by MisakiKata and David Miller.
Server Side Request Forgery via lxml parser. Reported by MisakiKata and David Miller.

Thank you to all who contacted the Plone security team to report problems!

Version support: The hotfix is officially supported by the Plone security team on the following versions of Plone in accordance with the Plone version support policy: 4.3.20, 5.0.10, 5.1.7, 5.2.4. Previous versions, like 4.2, could be affected but have not been tested. On such old versions, the hotfix might have worse side effects than what it tries to fix.

The fixes included here will be incorporated into subsequent releases of Plone, so Plone 5.2.5 and greater should not require this hotfix.

Warning: The hotfix has not been tested with Python 2.6. Originally Plone 4.3 was supported on Python 2.6, but since a few releases this is no longer the case since. It gets ever more difficult to test on Python 2.6. By now, you may have trouble installing any package with Python 2.6.

Zope support:

Zope is also affected. New versions for Zope and other packages are available. Upgrading to those is the recommended way.

If you cannot upgrade yet, you can try the Plone hotfix. It has not been tested on Zope only, but we try not to let the Plone-specific code get in the way, so it should be okay.

These vulnerabilities mentioned above are relevant for Zope:

Remote Code Execution via traversal in expressions via aliases.
Fixes released in Zope 4.6 and 5.2.
Remote Code Execution via traversal in expressions (no aliases).
Fixes released in Zope 4.6.1 and 5.2.1.
Various information disclosures.
Fixes released in Products.PluggableAuthService 2.6.0, Products.GenericSetup 2.1.1, and Zope 4.5.5.
Reflected XSS in various spots.
Fixes released in Products.CMFCore 2.5.1 and Products.PluggableAuthService 2.6.2.

The patch was released at 2021-05-18 15:00 UTC.

Installation

Full installation instructions are available on the HotFix release page.

Standard security advice

Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
Use an intrusion detection system that monitors key system resources for unauthorized changes.
Monitor your Zope, reverse-proxy request and system logs for unusual activity.
Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

If you do not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team released the patch at 2021-05-18 15:00 UTC.

Q. What will be involved in applying the patch?
A. Patches are made available as tarball-style archives that may be unpacked into the products folder of a buildout installation (for Plone 5.1.x and earlier only) and as Python packages that may be installed by editing a buildout configuration file and running buildout. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

Security vulnerability pre-announcement: 20210518

Posted by PLONE.ORG on May 05, 2021 11:00 PM

This is a routine patch with our standard 14 day notice period. There is no evidence that the issues fixed here are being used against any sites.

CVE numbers not yet issued.

Versions Affected: All supported Plone versions: 4.3, 5.0, 5.1 and 5.2. Previous versions could be affected but have not been tested.

Versions Not Affected: None.

Zope: Zope is also affected. A new version of Zope, or a related package, will be available around the time of the hotfix. If you cannot upgrade to this new version, you can try the Plone hotfix on your Zope site.

Nature of vulnerability: High, possible data exposure and remote code execution for already privileged users, various XSS (Cross Site Scripting) vulnerabilities.

The patch will be released at 2021-05-18 15:00 UTC.

Preparation

This is a pre-announcement of availability of this security fix.

The security fix egg will be named Products.PloneHotfix20210518 and its version will be 1.0. Further installation instructions will be made available when the fix is released.

Standard security advice

Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
Use an intrusion detection system that monitors key system resources for unauthorized changes.
Monitor your Zope, reverse-proxy request and system logs for unusual activity.
Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

Should you not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team will release the patch at 2021-05-18 15:00 UTC.

Q. What will be involved in applying the patch?
A. Patches are made available as Python packages that may be installed by editing a buildout configuration file and running buildout. For Plone 5.1 and lower they are also available as tarball-style archives that may be unpacked into the products folder of a buildout installation. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

Q: How can I help the Plone Security Team?

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

World Plone Day 2021 - Over 50 Videos from 16 Countries

Posted by PLONE.ORG on May 04, 2021 03:34 PM

World Plone Day, held on April 28th 2021, was a worldwide 24-hour online streaming event. The goal was to promote and educate the public about the benefits of using Plone and being part of the Plone community.

The event was a massive success! The amazing Plone community produced 56 videos totaling 22 hours of content, now available on our YouTube channel. More than 50 speakers from 16 countries presented case studies, tech tips and community insights in 11 languages. This includes an introduction to Plone 6 in EN, DE, NL, CA, IT, PT-BR, FI, and soon JP, ES, and EU.

How to Access World Plone Day Videos

All content is available on the Plone YouTube channel. Play the videos on the World Plone Day 2021 playlist to relive the entire event.

Remember to subscribe while you are there!

Highlights

General Interest

Plone 6

Technical Talks

Case Studies

...and if you speak Italian

World Plone Day in Italy features 3 hours of topics!

Follow Plone and Join the Community!

Stay up to date with Plone and join us:

Follow Plone on Twitter at twitter.com/plone
Subscribe to the Plone YouTube channel
Join the community forum at community.plone.org/
Get started with Plone at plone.org/get-started

On The Road to Plone 6 - Plone REST API 7 and Volto 12 Released

Posted by PLONE.ORG on April 23, 2021 07:16 PM

Here are two important releases on the road to Plone 6:

Plone REST API 7 introduces a new link-integrity feature for blocks-based pages.
Volto 12 improves the add-on ecosystem by introducing a new configuration registry to avoid circular dependencies.

This is a short version of a full blog post at Kitconcept site.

Plone REST API 7

Keeping links within a website intact is one of the core features of any Content Management System. In Plone, editors can copy and move single pages as well as large content trees without breaking internal links to other parts of the site. This is accomplished by using unique IDs (UUIDs) instead of relative or absolute paths when adding a link to another page.

Volto introduces blocks-based page layouts that store a JSON structure internally instead of HTML. This more structured way of storing page content and layout information allows more complex page layouts. Though, because of this change, the existing portal-transforms mechanism that rewrites links to UUIDs did not work any longer.

Many participants joined to improve this feature in a community-wide effort:

Werkbank, a Plone agency from Bochum, Germany, stepped up to sponsor the development of link integrity in 2020
Timo Stollenwerk from Kitconcept started to draft a possible solution and wrote the first prototype
Thomas Buchenberger from 4teamwork picked up that work at the Plone Conference sprint in Ferrara, Italy
Andrea Cecchi from RedTurtle joined the effort and refactored the resolveUID algorithm into a blocks transformer that made the resolveUID transformer more generic and flexible
After that, a first plone.restapi 7 alpha was released and entered a period of quality assurance and testing

After the first alpha release, the ResolveUID transformation was added to links and images. In addition, plone.restapi 7 comes with a new blocks serialization mechanism and an important fix that makes sure files are opened directly by Plone for anonymous users.

Additionally, a "smart fields" concept allows integrators to mark a blocks field as searchableText.

Also, a new @contextnavigation endpoint was added that allows for local navigations.

Plone REST API 7 was included in Plone 5.2.4 release.

Volto 12

A new Volto configuration registry is the new central point to store and retrieve Volto configurations. The configuration registry ensures a setting only exists once.

You can find more details about it in the Volto docs and in the Volto 12 upgrade guide.

Volto is the new React front end for Plone, communicating with the back end through the RestAPI.

Road to Plone 6

Plone REST API 7 and Volto 12 are two very important releases on the road to Plone 6. Next will come another Plone REST API 8 branch and release for Plone 6 supporting Python 3 only.

Volto will continue to move at a very high pace towards Plone 6. A series of Plone 6 “Micro-Sprints” will be organized to push things further.

Check out the Volto Roadmap on Github for more details.

Questions for the April Steering Circle?

Posted by PLONE.ORG on April 16, 2021 02:00 PM

As described in the Foundation's July discussion of Plone governance, a series of Steering Circle meetings is being held to discuss issues with our organizational structure and processes. This is part of the Foundation's initiative to solicit ideas for changes that will better serve the needs of our community, our projects, and our teams. The meetings will be held every two months, and the next one will be April 20th at 3:00 PM UTC. Each Plone team will send one or two representatives, including the Zope, Volto, RestAPI and Guillotina teams.

The Steering Circle meetings will include a discussion of questions from community members. Please use this form to submit any questions you have and we will put them on the agenda.

Thank you for being an awesome community and for helping to move Plone forward into its third decade!

On The Road to Plone 6 - Plone REST API 7 and Volto 12 Released!

Posted by kitconcept GmbH on April 12, 2021 09:24 AM

Two weeks ago we cut two important releases on the road to Plone 6. Plone REST API 7 introduces a new link integrity feature for blocks-based pages. Volto 12 improves the add-on ecosystem by introducing a new configuration registry to avoid circular dependencies.

Photo by Stephen Leonardi on Unsplash

Plone REST API 7

Linking pages is one of the core idea of the world wide web. Keeping links within a website intact is therefore one of the core features that any Content Management System needs to provide.

One of the core features of Plone has always been that editors can copy and move single pages as well as large content trees without breaking internal links to other parts of the site. This is accomplished by using unigue IDs (UUIDs) instead of relative or absolute paths when adding a link to another page. Plone uses “portal transforms” internally to rewrite those links in RichText fields on save operations.

Plone 6 (aka Volto) introduces blocks-based page layouts, that store a JSON structure internally instead of HTML. This more structured way of storing page content and layout allows more complex page layouts. Though, because of this change, the existing portal transforms mechanism that rewrites links to UUIDs did not work any longer.

Blocks-based Volto page in edit mode (with kitconcept-blocks-grid)

Werkbank, a Plone agency from Bochum, Germany, stepped up to sponsor the development of link integrity in 2020, since they needed that for a client project.

I started to draft a possible solution and wrote a first prototype. Thomas Buchenberger from 4teamwork picked up that work at the Plone Conference sprint in Ferrara, Italy.

Andrea Cecchi from RedTurtle joined our efforts and refactored the resolveUID algorithm into a blocks transformer, that we started to use for other use cases and that made the resolveUID transformer more generic and flexible.

After that, we cut a first plone.restapi 7 alpha release and entered a longer period of quality assurance and testing.

Two weeks ago, after we tested the new feature in multiple projects at kitconcept, I cut a final release of Plone REST API 7.

After the first alpha release, the ResolveUID transformation was added to links and images. In addition plone.restapi 7 comes with a new blocks serialization mechanism and an important fix that makes sure files are opened directly by Plone for anonymous users.

We also added a “smart fields” concept that allows integrators to mark a blocks field as searchableText field.

A new @contextnavigation endpoint was added that allows for local navigations. We enhanced the navigation endpoint to expose an optional navigation title (“nav_title”) field.

Since the final 7 release, we cut six more releases and REST API 7 was included in Plone 5.2.4 release.

Volto 12

Photo by Gaelle Marcel on Unsplash

Tiberiu and Victor worked on a new Volto configuration registry which is the new central point to store and retrieve Volto configurations. The configuration registry is a singleton that ensures a setting only exists once. With more an more Volto add-ons that started to depend on each other, we started to run into circular dependencies issues that we can avoid now with the new configuration registry.

You can find more details about it in the Volto docs and in the Volto 12 upgrade guide.

Plone 6

Plone REST API 7 and Volto 12 were two very important releases on our road to Plone 6.

We plan to cut another Plone REST API 8 branch and release for Plone 6, which will support Python 3 only.

Volto will continue to move at very high pace towards Plone 6. We started to organize a series of Plone 6 “Micro-Sprints” to push things further.

Check out the Volto Roadmap on github for more details.

Plone Training Is Now Available Online

Posted by PLONE.ORG on April 09, 2021 10:08 PM

When the pandemic forced the annual Plone Conference online, the conference training classes went online too. We sure did miss seeing everyone in person, but there's a small silver lining. The online training sessions were all recorded and they have now been posted to the Plone YouTube channel. Now anyone can watch and re-watch them!

The following videos are available. All are about 4 hours long.

Mastering Plone 6 - Part 1

Philip Bauer, Instructor

Learn how to develop custom projects with Plone 6 and Volto, the new React-based frontend. Covers the core technologies involved in Plone 6 programming, including how to write your own add-on package and customize your Plone site by writing Python code and React components. Covers the first half of the material in training.plone.org/5/mastering-plone.

Mastering Plone 6 - Part 2

Philip Bauer, Instructor

A continuation of Part 1. Covers the second half of the material in training.plone.org/5/mastering-plone.

React and Volto - Part 1: React

Jakob Kahl and Alok Kumar, Instructors

Get started with React so you can create your own site using Volto. Learn the basis of React, Redux and React-Router. Covers the material in training.plone.org/5/react/index.html.

React and Volto - Part 2: Volto

Jakob Kahl and Alok Kumar, Instructors

Learn how to how to quickly bootstrap and customize a Volto project, and how create your own website based on Volto. Covers the material in training.plone.org/5/volto/index.html and training.plone.org/5/voltohandson/index.html.

Volto Add-ons - Part 1

Tiberiu Ichim and Víctor Fernández de Alba, Instructors

Learn how to develop Volto add-ons and other useful Volto patterns. Learn how to quickly develop a real world Volto add-on and how to structure your code to make it simple, reusable and provide extensible components. Covers the material in training.plone.org/5/voltoaddons/index.html.

Volto Add-ons - Part 2

Tiberiu Ichim and Víctor Fernández de Alba, Instructors

A continuation of Part 1. Covers the material in training.plone.org/5/voltoaddons/index.html.

Getting Started With Your Plone Site

David Bain, Instructor

Content management principles for Plone. A little background, principles & concepts, logging in & out, preferences and password management, folder management and the basic publication workflow.

Pyramid

Steve Piercy, Instructor

A hands-on, quick tutorial covering "a little about a lot". Practical introductions to the most common features. Fun, fast-paced, and targeted to newcomers to both Python web application development and the Pyramid web framework.

Implementing user epics with BPMN

Posted by Asko Soukka on March 21, 2021 12:00 PM

Many user stories start simple. For example: “A user self registers into a course”. The final version, of course, tend to be more like: “A new user, after being verified to be an authentic real person, creates their new user account, immediately forgets and resets their new password, finally logs into the system, finds the course again, completes payment for the registration, and is then eventually enrolled into the course.” This is no longer a user story, but a series of stories supposed to be completed in a specific sequence. Let’s call these user epics.

Individual user stories, by their definition, should be relatively straightforward to implement. For example, a stereotypical user story may require interaction with specific user interface element to start the story, following one or more views to gather the required user input to continue the story, finally ending with user’s goal for the story. Where did the user come from, or where does the user continue afterwards, is simply not part of the story. On purpose.

But user epics are different.

An implementation of a user epic must know where the user came from, and must be able to decide, which stories the user should be guided to complete next. Epic should also cover the consequences of user failing to complete a story, or completely abandoning their fate. Therefore implementing an epic is not only about managing state, but also about programming the often complex business logic. Both being among the least fun parts of software development.

As an insult to injury, less and less are users doing their work alone, more and more they are being accompanied by various forms of automation. So, not only should an implemented epic be aware of the actions of its user, but it should also be able to react on known results or side effects of automation – anything from synchronous function calls to asynchronously executed software robots.

With just code, implementations for this kind of epics could get dirty and hard to understand, fast.

Offloading state and logic to BPMN engine

It’s not a coincidence that my example user epics above are drawn as BPMN diagrams. That’s what I have been doing for the last couple of months in our latest project. We had to modernize a process, where users were found to require a lot of hand-holding to be able complete all the required steps and in the right order.

Also, with the previous implementation, when users got lost in the middle of their epics, or an unexpected system error blocked them, it was often surprisingly hard to figure out what really had happened.

This time all this should be different.

Whenever a user starts a new epic, a new BPMN modeled process is being started. The model may be as long and complex as needed. It may even be a composition of multiple nested models, resulting in an arbitrarily deep process tree when executed. But this is not a problem, because this is exactly what dedicated BPMN process execution engines have been built for.

The fun part: With a BPMN engine executing our user epics directly from their BPMN models, we don’t really need to “implement” them at all. Instead, we:

Implement user interface routes and views for individual user tasks.
Make the views to read their preconditions from the process engine state.
Eventually complete the task at the engine (with user input when required).
Poll the engine for the next available task and redirect the user there.

Now our user task views are standalone, with no magical dependencies between them to code and maintain.

Plone, Volto and Camunda FTW

I would not be writing this, unless all the above could not be implemented by using open source software only:

Because every application seems to need a CMS (sooner or later), we build on top of Plone, the open source CMS we trust. Thanks to its powerful and complete REST API, we can do all the integrations we need with just HTTP requests – no need to know about Plone internals.
For modern user experience, we use Volto – ReactJS based web experience on top of Plone REST API (with Razzle based server side rendering support). Every part of Volto is easily customizable and extensible. For example, custom server side routes, like the ones we need to authorize and proxy calls to BPMN engine, are common and well documented use case.
Finally, for the BPMN engine, we choose Camunda BPM Community Edition. While the open source version of Camunda is probably missing countless convenient features of their enterprise offering, its BPMN engine and REST API come uncompromised.

Thanks to all above, it’s not a (technical) problem at all that users tend to forget their passwords just after sign up. Now the process engine remembers were they came from, and what they were supposed to do, just after resetting their password one more time…

Plone Connection Podcast: Episode 01 - Philip Bauer

Posted by Starzel.de on November 19, 2020 04:56 PM

The Plone Connection Podcast is a monthly podcast produced by Six Feet Up. Every month, Six Feet Up's Director of Engineering T. Kim Nguyen sits down with a different member of the Plone Community and asks them about their work with or on the Plone CMS.

Many thank to my good friend Kim for doing this!

Your first Plone 6 Project

Posted by Starzel.de on November 18, 2020 01:25 PM

I've had the opportunity to give this talk at the most excellent Python Web Conference.

ZODB Database debugging

Posted by Starzel.de on August 24, 2020 11:00 AM

The problem

The ZODB contains python objects serializes as pickles. When a object is loaded/used a pickle is deserialized ("unpickled") into a python object.

A ZODB can contain objects that cannot be loaded. Reasons for that may be:

Code could not be loaded that is required to unpickle the object (e.g. removed packages, modules or classes)
Objects are referenced but missing from the database (e.g. a blob is missing)
The objects contains invalid entries (e.g. a reference to a oid that uses a no longer supported format)

The most frequent issues are caused by:

Improperly uninstalled or removed packages (e.g. Archetypes, ATContentTypes, CMFDefault, PloneFormGen etc.)
Code has changed but not all objects that rely on that code as updated
Code was refactored and old imports are no longer working

You should not blame the migration to Python 3 for these issues! Many issues may already exists in their database before the migration but people usually do not run checks to find issues. After a migration to Python 3 most people check their database for the first time. This may be because the documentation on python3-migration recommends running the tool zodbverify.

Real problems may be revealed at that point, e.g when:

Packing the Database fails
Features fail

You can check you ZODB for problems using the package zodbverify. To solve each of the issues you need to be able to answer three questions:

Which object is broken and what is the error?
Where is the object and what uses it?
How do I fix it?

In short these approaches to fixing exist:

Ignore the errors
Add zodbupgrade mappings
Patch your python-path to work around the errors
Replace broken objects with dummies
Remove broken objects the hard way
Find our what and where broken objects are and then fix or remove them safely

I will mostly focus on the last approach.

But before you spend a lot of time to investigate individual errors it would be a good idea to deal with the most frequent problems, especially IntIds and Relations (see the chapter "Frequent Culprits") below. In my experience these usually solved most issues.

Find out what is broken

Check your entire database

Use zodbverify to verify a ZODB by iterating and loading all records. zodbverify is available as a standalone script and as addon for plone.recipe.zope2instance. Use the newest version!

In the simplest form run it like this:

$ bin/zodbverify -f var/filestorage/Data.fs

It will return:

a list of types of errors
the number of occurences
all oids that raise that error on loading

Note

zodbverify is only available for Plone 5.2 and later. For older Plone-Versions use the scripts fstest.py and fsrefs.py from the ZODB package:

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fstest.py var/filestorage/Data.fs

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fsrefs.py var/filestorage/Data.fs

The output of zodbverify might look like this abbreviated example from a medium-sized intranet (1GB Data.fs, 5GB blobstorage) that started with Plone 4 on Archetypes and was migrated to Plone 5.2 on Python 3 and Dexterity:

$ ./bin/zodbverify -f var/filestorage/Data.fs



[...]



INFO:zodbverify:Done! Scanned 163955 records.

Found 1886 records that could not be loaded.

Exceptions, how often they happened and which oids are affected:



ModuleNotFoundError: No module named 'Products.Archetypes': 1487

0x0e00eb 0x0e00ee 0x0e00ef 0x0e00f0 0x0e00f1 0x2b194b 0x2b194e 0x2b194f 0x2b1950 [...]



ModuleNotFoundError: No module named 'Products.PloneFormGen': 289

0x2b1940 0x2b1941 0x2b1942 0x2b1943 0x2b1944 0x2b1974 0x2b1975 0x2b1976 0x2b1977 [...]



AttributeError: module 'App.interfaces' has no attribute 'IPersistentExtra': 34

0x2c0a69 0x2c0a6b 0x2c0ab7 0x2c0ab9 0x2c555d [...] 0x35907f



ModuleNotFoundError: No module named 'Products.CMFDefault': 20

0x011e 0x011f 0x0120 0x0121 0x0122 0x0123 0x0124 0x0125 0x0126 0x0127 0x0128 0x0129 0x012a 0x012b 0x012c 0x012d 0x012e 0x012f 0x0130 0x0131



ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 20

0x3b1cde 0x3b1ce0 0x3b1ce4 0x3b1ce6 0x3b1ce9 0x3b1ceb 0x3b1cee 0x3b1cf0 0x3b1cf4 0x3b1cf6 0x3b1cf9 0x3b1cfb 0x3b1cfe 0x3b1d00 0x3b1d04 0x3b1d06 0x3b1d09 0x3b1d0b 0x3b1d0e 0x3b1d10



ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

0x2c0a68 0x2c0a6a 0x2c555c 0x2c555e 0x2c560b 0x2c560d 0x2c5663 0x2c5665 0x2c571b 0x2c571d 0x2c5774 0x2c5776 0x2c5833 0x2c5835 0x33272d 0x33272f



ModuleNotFoundError: No module named 'fourdigits': 8

0x28030f 0x280310 0x280311 0x280312 0x280313 0x280314 0x280315 0x280316



ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

0x0e00e9 0x0e011a 0x0e01b3 0x0e0cb3



AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

0x2a712b 0x2a712c 0x2a712d



ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

0x11



ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

0x25



ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

0x37



ModuleNotFoundError: No module named 'plone.app.controlpanel': 1

0x0f4c2b



ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

0x3b1311

You can see all different types of errors that appear and which objects are causing them. Objects are referenced by their oid in the ZODB. See the Appendix on how to deal with oids.

You can see that among other issues there are still a lot of references to Archetypes and PloneFormGen (I omitted the complete lists) even though both are no longer used in the site.

Before the summary the log dumps a huge list of errors that contain the pickle and the error:

INFO:zodbverify:

Could not process unknown record 0x376b77 (b'\x00\x00\x00\x00\x007kw'):

INFO:zodbverify:b'\x80\x03cProducts.PloneFormGen.content.thanksPage\nFormThanksPage\nq\x00.\x80\x03}q\x01(X\x0c\x00\x00\x00showinsearchq\x02\x88X\n\x00\x00\x00_signatureq\x03C\x10\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7q\x04X\r\x00\x00\x00creation_dateq\x05cDateTime.DateTime\nDateTime\nq\x06)\x81q\x07GA\xd6\xdf_\xba\xd56"\x89X\x05\x00\x00\x00GMT+2q\x08\x87q\tbX\r\x00\x00\x00marshall_hookq\nNX\n\x00\x00\x00showFieldsq\x0b]q\x0cX\x02\x00\x00\x00idq\rX\t\x00\x00\x00thank-youq\x0eX\x11\x00\x00\x00_at_creation_flagq\x0f\x88X\x11\x00\x00\x00modification_dateq\x10h\x06)\x81q\x11GA\xd6\xdf_\xba\xd7\x15r\x89h\x08\x87q\x12bX\x05\x00\x00\x00titleq\x13X\x05\x00\x00\x00Dankeq\x14X\x0f\x00\x00\x00demarshall_hookq\x15NX\x0e\x00\x00\x00includeEmptiesq\x16\x88X\x0e\x00\x00\x00thanksEpilogueq\x17C\x08\x00\x00\x00\x00\x007k\xaaq\x18cProducts.Archetypes.BaseUnit\nBaseUnit\nq\x19\x86q\x1aQX\x07\x00\x00\x00showAllq\x1b\x88X\x12\x00\x00\x00_EtagSupport__etagq\x1cX\r\x00\x00\x00ts34951147.36q\x1dX\x0b\x00\x00\x00portal_typeq\x1eX\x0e\x00\x00\x00FormThanksPageq\x1fX\x0b\x00\x00\x00searchwordsq C\x08\x00\x00\x00\x00\x007k\xabq!h\x19\x86q"QX\x07\x00\x00\x00_at_uidq#X \x00\x00\x00a2d15a36a521471daf2b7005ff9dbc62q$X\r\x00\x00\x00at_referencesq%C\x08\x00\x00\x00\x00\x007k\xacq&cOFS.Folder\nFolder\nq\'\x86q(QX\x0e\x00\x00\x00thanksPrologueq)C\x08\x00\x00\x00\x00\x007k\xadq*h\x19\x86q+QX\x0f\x00\x00\x00noSubmitMessageq,C\x08\x00\x00\x00\x00\x007k\xaeq-h\x19\x86q.QX\x03\x00\x00\x00_mdq/C\x08\x00\x00\x00\x00\x007k\xafq0cPersistence.mapping\nPersistentMapping\nq1\x86q2QX\x12\x00\x00\x00__ac_local_roles__q3}q4X\x16\x00\x00\x00xxx@xxx.deq5]q6X\x05\x00\x00\x00Ownerq7asu.'

INFO:zodbverify:Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 62, in verify_record

    class_info = unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'Products.PloneFormGen'

Inspecting a single object

In this case the object with the oid 0x376b77 seems to be a FormThanksPage from Products.PloneFormGen. But wait! You deleted all of these, so where in the site is it?

If the offending object is normal content the solution is mostly simple. You can call obj.getPhysicalPath() to find out where it is. But ore often than not editing and saving will fix the problem. In other cases you might need to copy the content to a new item and delete the broken object.

But usually it is not simply content but something else. Here are some examples:

A annotation on a object or the portal
A relationvalue in the relatopn-catalog
A item in the IntId catalog
A old revision of content in CMFEditions
A configuration-entry in portal_properties or in portal_registry

The hardest part is to find out what and where the broken object actually is before removing or fixing it.

The reason for that is that a entry in the ZODB does not know about it's parent. Acquisition finds parents with obj.aq_parent() but many items are not-Acquisition-aware. Only the parents that reference objects know about them.

A object x could be the attribute some_object on object y but you will not see that by inspecting x. Only y knows that x is y.some_object.

A way to work around this is used by the script fsoids.py on ZODB. It allows you to list all incoming and outgoing references to a certain object.

With this you will see that x is referenced by y. With this information you can then inspect the object y and hopefully see how x is set on y.

More often than not y is again not a object in the content-hierarchy but maybe a BTree of sorts, a pattern that is frequently used for effective storage of many items. Then you need to find out the parent of y to be able to fix x.

And so forth. It can a couple of steps until you end up in a item that can be identified, e.g. portal_properties or RelationCatalog and usually only exists once in a database.

To make the process of finding this path less tedious I extended zodbverify in https://github.com/plone/zodbverify/pull/8 with a feature that will show you all parents and their parents in a way that allows you to see where in the tree is it.

Before we look at the path of 0x376b77 we'll inspect the object.

Pass the oid and the debug-flag -D to zodbverify with ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D:

$ ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D



INFO:zodbverify:Inspecting 0x376b77:

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

INFO:zodbverify:

Object as dict:

{'__Broken_newargs__': (), '__Broken_state__': {'showinsearch': True, '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7', 'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'), 'marshall_hook': None, 'showFields': [], 'id': 'thank-you', '_at_creation_flag': True, 'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'), 'title': 'Danke', 'demarshall_hook': None, 'includeEmpties': True, 'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>, 'showAll': True, '_EtagSupport__etag': 'ts34951147.36', 'portal_type': 'FormThanksPage', 'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>, '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62', 'at_references': <Folder at at_references>, 'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>, 'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>, '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>, '__ac_local_roles__': {'xxx@xxx.de': ['Owner']}}}

INFO:zodbverify:

The object is 'obj'

[2] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(118)verify_oid()

-> pickle, state = storage.load(oid)

(Pdb++)

Even before you use the provided pdb to inspect it you can see that it is of the class persistent broken, a way of the ZODB to give you access to objects even though their class can no longer be imported.

You can now inspect it:

(Pdb++) obj

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

(Pdb++) pp obj.__dict__

{'__Broken_newargs__': (),

 '__Broken_state__': {'_EtagSupport__etag': 'ts34951147.36',

                      '__ac_local_roles__': {'xx@xxx.de': ['Owner']},

                      '_at_creation_flag': True,

                      '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62',

                      '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>,

                      '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7',

                      'at_references': <Folder at at_references>,

                      'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'),

                      'demarshall_hook': None,

                      'id': 'thank-you',

                      'includeEmpties': True,

                      'marshall_hook': None,

                      'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'),

                      'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>,

                      'portal_type': 'FormThanksPage',

                      'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>,

                      'showAll': True,

                      'showFields': [],

                      'showinsearch': True,

                      'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>,

                      'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>,

                      'title': 'Danke'}}

If you now choose to continue (by pressing c) zodbverify it will try to disassemble the pickle. That is very useful for in-depth debugging but out of the scope of this documentation.

Inspect the path of references

Now you know it is broken but you still don't know where this ominous FormThanksPage actually is.

Continue to let zodbverify find the path to the object:

INFO:zodbverify:Building a reference-tree of ZODB...

[...]

INFO:zodbverify:Created a reference-dict for 163955 objects.



INFO:zodbverify:

This oid is referenced by:



INFO:zodbverify:0x376ada BTrees.IOBTree.IOBucket at level 1

INFO:zodbverify:0x28018c BTrees.IOBTree.IOBTree at level 2

INFO:zodbverify:0x280184 five.intid.intid.IntIds at level 3

INFO:zodbverify:0x1e five.localsitemanager.registry.PersistentComponents at level 4

INFO:zodbverify:0x11 Products.CMFPlone.Portal.PloneSite at level 5

INFO:zodbverify:0x01 OFS.Application.Application at level 6

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f6 persistent.mapping.PersistentMapping at level 7

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at level 8

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f5 plone.app.redirector.storage.RedirectionStorage at level 6

INFO:zodbverify:0x02fa zope.ramcache.ram.RAMCache at level 7

INFO:zodbverify:0x02fd plone.contentrules.engine.storage.RuleStorage at level 8

INFO:zodbverify:0x338f13 plone.app.contentrules.rule.Rule at level 9

INFO:zodbverify:0x0303 BTrees.OOBTree.OOBTree at level 10

INFO:zodbverify:0x346961 plone.app.contentrules.rule.Rule at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x02fe plone.app.viewletmanager.storage.ViewletSettingsStorage at level 9

INFO:zodbverify:0x034d plone.keyring.keyring.Keyring at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x376864 BTrees.IOBTree.IOBucket at level 3

INFO:zodbverify:0x31049f BTrees.IOBTree.IOBucket at level 4

INFO:zodbverify:0x325823 BTrees.IOBTree.IOBucket at level 5

INFO:zodbverify:0x3984c8 BTrees.IOBTree.IOBucket at level 6

INFO:zodbverify:0x2cce9a BTrees.IOBTree.IOBucket at level 7

INFO:zodbverify:0x2c6669 BTrees.IOBTree.IOBucket at level 8

INFO:zodbverify:0x2c62b4 BTrees.IOBTree.IOBucket at level 9

INFO:zodbverify:0x2c44c1 BTrees.IOBTree.IOBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x377536 BTrees.OIBTree.OIBucket at level 2

INFO:zodbverify:0x376b14 BTrees.OIBTree.OIBucket at level 3

INFO:zodbverify:0x376916 BTrees.OIBTree.OIBucket at level 4

INFO:zodbverify:0x376202 BTrees.OIBTree.OIBucket at level 5

INFO:zodbverify:0x373fa7 BTrees.OIBTree.OIBucket at level 6

INFO:zodbverify:0x37363a BTrees.OIBTree.OIBucket at level 7

INFO:zodbverify:0x372f26 BTrees.OIBTree.OIBucket at level 8

INFO:zodbverify:0x372cc8 BTrees.OIBTree.OIBucket at level 9

INFO:zodbverify:0x36eb86 BTrees.OIBTree.OIBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x407185 BTrees.OIBTree.OIBTree at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

You can see from the logged messages that the FormThanksPage is in a IOBucket which again is in a IOBTree which is in a object of the class five.intid.intid.IntIds which is part if the component-registry in the Plone site.

This means there is a reference to a broken object in the IntId tool. How to solve all these is covered below in the chapter "Frequent Culprits".

Decide how and if to fix it

In this case the solution is clear (remove refs to broken objects from the intid tool). But that is only one approach.

Often the solution is not presented like this (the solution to intid was not obvious to me until I spent considerable time to investigate).

The following six options to deal with these problems exists. Spoiler: Option 6 is the best approach in most cases but the other also have valid use-cases.

Option 1: Ignoring the errors

I do that a lot. Especially old databases that were migrated all the may from Plone 2 or 3 up to the current version have issues. If these issues never appear during operation and if clients have no budget or interest in fixing them you can leave them be. If they do not hurt you (e.g. you cannot pack your database or features actually fail) you can choose to ignore them.

At some point later they might appear and it may be a better time to fix them. I spent many hours fixing issues that will never show during operation.

Option 2: Migrating/Fixing a DB with zodbupdate

Use that when a module or class has moved or was renamed.

Docs: https://github.com/zopefoundation/zodbupdate

You can change objects in DB according to rules:

When a import has moved use a rename mapping
To specify if a obj needs to be decoded decode mapping

Examples from Zope/src/OFS/__init__.py:

zodbupdate_decode_dict = {

    'OFS.Image File data': 'binary',

    'OFS.Image Image data': 'binary',



    'OFS.Application Application title': 'utf-8',

    'OFS.DTMLDocument DTMLDocument title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod raw': 'utf-8',

    'OFS.Folder Folder title': 'utf-8',

    'OFS.Image File title': 'utf-8',

    'OFS.Image Image title': 'utf-8',

    'OFS.Image Pdata title': 'utf-8',

    'OFS.Image Pdata data': 'binary',

    'OFS.OrderedFolder OrderedFolder title': 'utf-8',

    'OFS.userfolder UserFolder title': 'utf-8',

}



zodbupdate_rename_dict = {

    'webdav.LockItem LockItem': 'OFS.LockItem LockItem',

}

You can specify your own mappings in your own packages. These mappings need to be registered in setup.py so zodbupdate will pick them up.

Rename mapping example: https://github.com/zopefoundation/Zope/commit/f677ed7

Decode mapping example: https://github.com/zopefoundation/Products.ZopeVersionControl/commit/138cf39

Option 3: Work around with a patch

You can inject a module to work around missing or moved classes or modules.

The reason to want do this is usually because then you can safely delete items after that. They don't hurt your performance.

Examples in __init__.py:

# -*- coding: utf-8 -*-

from OFS.SimpleItem import SimpleItem

from plone.app.upgrade.utils import alias_module

from plone.app.upgrade import bbb

from zope.interface import Interface





class IBBB(Interface):

    pass





class BBB(object):

    pass





SlideshowDescriptor = SimpleItem





# Interfaces

try:

    from collective.z3cform.widgets.interfaces import ILayer

except ImportError:

    alias_module('collective.z3cform.widgets.interfaces.ILayer', IDummy)





try:

    from App.interfaces import IPersistentExtra

except ImportError:

    alias_module('App.interfaces.IPersistentExtra', IDummy)





try:

    from webdav.interfaces import IDAVResource

except ImportError:

    alias_module('webdav.interfaces.IDAVResource', IDummy)





# SimpleItem

try:

    from collective.easyslideshow.descriptors import SlideshowDescriptor

except ImportError:

    alias_module('collective.easyslideshow.descriptors.SlideshowDescriptor', SlideshowDescriptor)





# object

try:

    from collective.solr import interfaces

except ImportError:

    alias_module('collective.solr.indexer.SolrIndexProcessor', BBB)





try:

    from Products.CMFPlone import UndoTool

except ImportError:

    sys.modules['Products.CMFPlone.UndoTool'] = bbb

More: https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/patches.py

Plone has plenty of these (see https://github.com/plone/plone.app.upgrade/blob/master/plone/app/upgrade/__init__.py)

Option 4: Replace broken objects with a dummy

If a objects is missing (i.e. you get a POSKeyError) or broken beyond repair you can choose to replace it with a dummy.

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    dummy = Persistent()

    dummy._p_oid = p64(oid)

    dummy._p_jar = app._p_jar

    app._p_jar._register(dummy)

    app._p_jar._added[dummy._p_oid] = dummy

transaction.commit()

You shoud be aware that the missing or broken object will be gone forever after you didi this. So before you choose to go down this path you should try to find out what the object in question actually was.

Option 5: Remove broken objects from db

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    root = connection.root()

    del app._p_jar[p64(oid)]

transaction.commit()

I'm not sure if that is a acceptable approach under any circumstance since this will remove the pickle but not all references to the object. It will probably lead to PosKeyErrors.

Option 6: Manual fixing

This is how you should deal with most problems.

The way to go

Use zodbverify to get all broken objects
Pick one error-type at a time
Use zodbverify with -o <OID> -D to inspect one object and find out where that object is referenced
If you use fsoids.py follow referenced by until you find where in the tree the object lives. zodbverify will try to do it for you.
Remove or fix the object (using a upgrade-step, pdb or a rename mapping)

Find out which items are broken

The newest version of zodbverify has a feature to that does the same task we discussed in Example 1 for you. Until it is merged and released you need to use the branch show_references from the pull-request https://github.com/plone/zodbverify/pull/8

When inspecting a individual oid zodbverify builds a dict of all references for reverse-lookup. Then it recursively follow the trail of references to referencing items up to the root. To prevent irrelevant and recursive entries it aborts after level 600 and at some root-objects because these usually references a lot and would clutter the result with irrelevant information.

The output should give you a pretty good idea where in the object-tree a item is actually located, how to access and fix it.

If 0x3b1d06 is the broken oid inspect it with zodbverify:

$ ./bin/instance zodbverify -o 0x3b1d06 -D



2020-08-24 12:19:32,441 INFO    [Zope:45][MainThread] Ready to handle requests

2020-08-24 12:19:32,442 INFO    [zodbverify:222][MainThread]

The object is 'obj'

The Zope instance is 'app'

[4] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(230)verify_oid()

-> pickle, state = storage.load(oid)



(Pdb++) obj

<BTrees.OIBTree.OITreeSet object at 0x110b97ac0 oid 0x3b1d06 in <Connection at 10c524040>>



(Pdb++) pp [i for i in obj]

[<InterfaceClass OFS.EtagSupport.EtagBaseInterface>,

 [...]

 <class 'webdav.interfaces.IDAVResource'>,

 <InterfaceClass plone.dexterity.interfaces.IDexterityContent>,

 <InterfaceClass plone.app.relationfield.interfaces.IDexterityHasRelations>,

 [...]

 <SchemaClass plone.supermodel.model.Schema>]

The problem now is that obj has no __parent__ so you have no way of knowing what you're actually dealing with.

When you press c for continue zodbverify will proceed and load the pickle:

(Pdb++) c

2020-08-24 12:20:50,784 INFO    [zodbverify:68][MainThread]

Could not process <class 'BTrees.OIBTree.OITreeSet'> record 0x3b1d06 (b'\x00\x00\x00\x00\x00;\x1d\x06'):

2020-08-24 12:20:50,784 INFO    [zodbverify:69][MainThread] b'\x80\x03cBTrees.OIBTree\nOITreeSet\nq\x00.\x80\x03(cOFS.EtagSupport\nEtagBaseInterface\nq\x01cAcquisition.interfaces\nIAcquirer\nq\x02cplone.app.dexterity.behaviors.discussion\nIAllowDiscussion\nq\x03czope.annotation.interfaces\nIAnnotatable\nq\x04czope.annotation.interfaces\nIAttributeAnnotatable\nq\x05cplone.uuid.interfaces\nIAttributeUUID\nq\x06cProducts.CMFDynamicViewFTI.interfaces\nIBrowserDefault\nq\x07cProducts.CMFCore.interfaces\nICatalogAware\nq\x08cProducts.CMFCore.interfaces\nICatalogableDublinCore\nq\tczope.location.interfaces\nIContained\nq\ncProducts.CMFCore.interfaces\nIContentish\nq\x0bcOFS.interfaces\nICopySource\nq\x0ccwebdav.interfaces\nIDAVResource\nq\rcplone.dexterity.interfaces\nIDexterityContent\nq\x0ecplone.app.relationfield.interfaces\nIDexterityHasRelations\nq\x0fcplone.dexterity.interfaces\nIDexterityItem\nq\x10cplone.app.iterate.dexterity.interfaces\nIDexterityIterateAware\nq\x11cplone.dexterity.interfaces\nIDexteritySchema\nq\x12cplone.app.contenttypes.interfaces\nIDocument\nq\x13cProducts.CMFCore.interfaces\nIDublinCore\nq\x14cProducts.CMFCore.interfaces\nIDynamicType\nq\x15cplone.app.dexterity.behaviors.exclfromnav\nIExcludeFromNavigation\nq\x16cz3c.relationfield.interfaces\nIHasIncomingRelations\nq\x17cz3c.relationfield.interfaces\nIHasOutgoingRelations\nq\x18cz3c.relationfield.interfaces\nIHasRelations\nq\x19cplone.namedfile.interfaces\nIImageScaleTraversable\nq\x1acOFS.interfaces\nIItem\nq\x1bcplone.app.iterate.interfaces\nIIterateAware\nq\x1ccplone.portlets.interfaces\nILocalPortletAssignable\nq\x1dczope.location.interfaces\nILocation\nq\x1ecOFS.interfaces\nIManageable\nq\x1fcProducts.CMFCore.interfaces\nIMinimalDublinCore\nq cProducts.CMFCore.interfaces\nIMutableDublinCore\nq!cProducts.CMFCore.interfaces\nIMutableMinimalDublinCore\nq"cplone.app.content.interfaces\nINameFromTitle\nq#cApp.interfaces\nINavigation\nq$cProducts.CMFCore.interfaces\nIOpaqueItemManager\nq%cAccessControl.interfaces\nIOwned\nq&cAccessControl.interfaces\nIPermissionMappingSupport\nq\'cpersistent.interfaces\nIPersistent\nq(cOFS.interfaces\nIPropertyManager\nq)cplone.app.relationfield.behavior\nIRelatedItems\nq*cAccessControl.interfaces\nIRoleManager\nq+cplone.contentrules.engine.interfaces\nIRuleAssignable\nq,cProducts.CMFDynamicViewFTI.interfaces\nISelectableBrowserDefault\nq-cOFS.interfaces\nISimpleItem\nq.cplone.app.contenttypes.behaviors.tableofcontents\nITableOfContents\nq/cOFS.interfaces\nITraversable\nq0cplone.uuid.interfaces\nIUUIDAware\nq1cProducts.CMFEditions.interfaces\nIVersioned\nq2cplone.app.versioningbehavior.behaviors\nIVersioningSupport\nq3cProducts.CMFCore.interfaces\nIWorkflowAware\nq4cOFS.interfaces\nIWriteLock\nq5cOFS.interfaces\nIZopeObject\nq6czope.interface\nInterface\nq7cplone.dexterity.schema.generated\nPlone_0_Document\nq8cplone.supermodel.model\nSchema\nq9tq:\x85q;\x85q<\x85q=.'

2020-08-24 12:20:50,786 INFO    [zodbverify:70][MainThread] Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 64, in verify_record

    unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package



    0: \x80 PROTO      3

    2: (    MARK

    3: c        GLOBAL     'OFS.EtagSupport EtagBaseInterface'

   38: q        BINPUT     1

   40: c        GLOBAL     'Acquisition.interfaces IAcquirer'

   74: q        BINPUT     2

   76: c        GLOBAL     'plone.app.dexterity.behaviors.discussion IAllowDiscussion'

  135: q        BINPUT     3

  137: c        GLOBAL     'zope.annotation.interfaces IAnnotatable'

  178: q        BINPUT     4

  180: c        GLOBAL     'zope.annotation.interfaces IAttributeAnnotatable'

  230: q        BINPUT     5

  232: c        GLOBAL     'plone.uuid.interfaces IAttributeUUID'

  270: q        BINPUT     6

  272: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces IBrowserDefault'

  327: q        BINPUT     7

  329: c        GLOBAL     'Products.CMFCore.interfaces ICatalogAware'

  372: q        BINPUT     8

  374: c        GLOBAL     'Products.CMFCore.interfaces ICatalogableDublinCore'

  426: q        BINPUT     9

  428: c        GLOBAL     'zope.location.interfaces IContained'

  465: q        BINPUT     10

  467: c        GLOBAL     'Products.CMFCore.interfaces IContentish'

  508: q        BINPUT     11

  510: c        GLOBAL     'OFS.interfaces ICopySource'

  538: q        BINPUT     12

  540: c        GLOBAL     'webdav.interfaces IDAVResource'

  572: q        BINPUT     13

  574: c        GLOBAL     'plone.dexterity.interfaces IDexterityContent'

  620: q        BINPUT     14

  622: c        GLOBAL     'plone.app.relationfield.interfaces IDexterityHasRelations'

  681: q        BINPUT     15

  683: c        GLOBAL     'plone.dexterity.interfaces IDexterityItem'

  726: q        BINPUT     16

  728: c        GLOBAL     'plone.app.iterate.dexterity.interfaces IDexterityIterateAware'

  791: q        BINPUT     17

  793: c        GLOBAL     'plone.dexterity.interfaces IDexteritySchema'

  838: q        BINPUT     18

  840: c        GLOBAL     'plone.app.contenttypes.interfaces IDocument'

  885: q        BINPUT     19

  887: c        GLOBAL     'Products.CMFCore.interfaces IDublinCore'

  928: q        BINPUT     20

  930: c        GLOBAL     'Products.CMFCore.interfaces IDynamicType'

  972: q        BINPUT     21

  974: c        GLOBAL     'plone.app.dexterity.behaviors.exclfromnav IExcludeFromNavigation'

 1040: q        BINPUT     22

 1042: c        GLOBAL     'z3c.relationfield.interfaces IHasIncomingRelations'

 1094: q        BINPUT     23

 1096: c        GLOBAL     'z3c.relationfield.interfaces IHasOutgoingRelations'

 1148: q        BINPUT     24

 1150: c        GLOBAL     'z3c.relationfield.interfaces IHasRelations'

 1194: q        BINPUT     25

 1196: c        GLOBAL     'plone.namedfile.interfaces IImageScaleTraversable'

 1247: q        BINPUT     26

 1249: c        GLOBAL     'OFS.interfaces IItem'

 1271: q        BINPUT     27

 1273: c        GLOBAL     'plone.app.iterate.interfaces IIterateAware'

 1317: q        BINPUT     28

 1319: c        GLOBAL     'plone.portlets.interfaces ILocalPortletAssignable'

 1370: q        BINPUT     29

 1372: c        GLOBAL     'zope.location.interfaces ILocation'

 1408: q        BINPUT     30

 1410: c        GLOBAL     'OFS.interfaces IManageable'

 1438: q        BINPUT     31

 1440: c        GLOBAL     'Products.CMFCore.interfaces IMinimalDublinCore'

 1488: q        BINPUT     32

 1490: c        GLOBAL     'Products.CMFCore.interfaces IMutableDublinCore'

 1538: q        BINPUT     33

 1540: c        GLOBAL     'Products.CMFCore.interfaces IMutableMinimalDublinCore'

 1595: q        BINPUT     34

 1597: c        GLOBAL     'plone.app.content.interfaces INameFromTitle'

 1642: q        BINPUT     35

 1644: c        GLOBAL     'App.interfaces INavigation'

 1672: q        BINPUT     36

 1674: c        GLOBAL     'Products.CMFCore.interfaces IOpaqueItemManager'

 1722: q        BINPUT     37

 1724: c        GLOBAL     'AccessControl.interfaces IOwned'

 1757: q        BINPUT     38

 1759: c        GLOBAL     'AccessControl.interfaces IPermissionMappingSupport'

 1811: q        BINPUT     39

 1813: c        GLOBAL     'persistent.interfaces IPersistent'

 1848: q        BINPUT     40

 1850: c        GLOBAL     'OFS.interfaces IPropertyManager'

 1883: q        BINPUT     41

 1885: c        GLOBAL     'plone.app.relationfield.behavior IRelatedItems'

 1933: q        BINPUT     42

 1935: c        GLOBAL     'AccessControl.interfaces IRoleManager'

 1974: q        BINPUT     43

 1976: c        GLOBAL     'plone.contentrules.engine.interfaces IRuleAssignable'

 2030: q        BINPUT     44

 2032: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces ISelectableBrowserDefault'

 2097: q        BINPUT     45

 2099: c        GLOBAL     'OFS.interfaces ISimpleItem'

 2127: q        BINPUT     46

 2129: c        GLOBAL     'plone.app.contenttypes.behaviors.tableofcontents ITableOfContents'

 2196: q        BINPUT     47

 2198: c        GLOBAL     'OFS.interfaces ITraversable'

 2227: q        BINPUT     48

 2229: c        GLOBAL     'plone.uuid.interfaces IUUIDAware'

 2263: q        BINPUT     49

 2265: c        GLOBAL     'Products.CMFEditions.interfaces IVersioned'

 2309: q        BINPUT     50

 2311: c        GLOBAL     'plone.app.versioningbehavior.behaviors IVersioningSupport'

 2370: q        BINPUT     51

 2372: c        GLOBAL     'Products.CMFCore.interfaces IWorkflowAware'

 2416: q        BINPUT     52

 2418: c        GLOBAL     'OFS.interfaces IWriteLock'

 2445: q        BINPUT     53

 2447: c        GLOBAL     'OFS.interfaces IZopeObject'

 2475: q        BINPUT     54

 2477: c        GLOBAL     'zope.interface Interface'

 2503: q        BINPUT     55

 2505: c        GLOBAL     'plone.dexterity.schema.generated Plone_0_Document'

 2556: q        BINPUT     56

 2558: c        GLOBAL     'plone.supermodel.model Schema'

 2589: q        BINPUT     57

 2591: t        TUPLE      (MARK at 2)

 2592: q    BINPUT     58

 2594: \x85 TUPLE1

 2595: q    BINPUT     59

 2597: \x85 TUPLE1

 2598: q    BINPUT     60

 2600: \x85 TUPLE1

 2601: q    BINPUT     61

 2603: .    STOP

highest protocol among opcodes = 2

If you are into this you can read the pickle now :)

If you press c again zodbverify will build the refernce-tree for this object and ispect if for you:

(Pdb++) c

2020-08-24 12:22:42,596 INFO    [zodbverify:234][MainThread] ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 0x3b1d06

2020-08-24 12:22:42,597 INFO    [zodbverify:43][MainThread] Building a reference-tree of ZODB...

2020-08-24 12:22:42,964 INFO    [zodbverify:60][MainThread] Objects: 10000

2020-08-24 12:22:44,167 INFO    [zodbverify:60][MainThread] Objects: 20000

2020-08-24 12:22:44,521 INFO    [zodbverify:60][MainThread] Objects: 30000

2020-08-24 12:22:44,891 INFO    [zodbverify:60][MainThread] Objects: 40000

2020-08-24 12:22:45,184 INFO    [zodbverify:60][MainThread] Objects: 50000

2020-08-24 12:22:45,507 INFO    [zodbverify:60][MainThread] Objects: 60000

2020-08-24 12:22:45,876 INFO    [zodbverify:60][MainThread] Objects: 70000

2020-08-24 12:22:46,403 INFO    [zodbverify:60][MainThread] Objects: 80000

2020-08-24 12:22:46,800 INFO    [zodbverify:60][MainThread] Objects: 90000

2020-08-24 12:22:47,107 INFO    [zodbverify:60][MainThread] Objects: 100000

2020-08-24 12:22:47,440 INFO    [zodbverify:60][MainThread] Objects: 110000

2020-08-24 12:22:47,747 INFO    [zodbverify:60][MainThread] Objects: 120000

2020-08-24 12:22:48,052 INFO    [zodbverify:60][MainThread] Objects: 130000

2020-08-24 12:22:48,375 INFO    [zodbverify:60][MainThread] Objects: 140000

2020-08-24 12:22:48,665 INFO    [zodbverify:60][MainThread] Objects: 150000

2020-08-24 12:22:48,923 INFO    [zodbverify:60][MainThread] Objects: 160000

2020-08-24 12:22:49,037 INFO    [zodbverify:61][MainThread] Created a reference-dict for 163955 objects.



2020-08-24 12:22:49,386 INFO    [zodbverify:182][MainThread] Save reference-cache as /Users/pbauer/.cache/zodbverify/zodb_references_0x03d7f331f3692266.json

2020-08-24 12:22:49,424 INFO    [zodbverify:40][MainThread] The oid 0x3b1d06 is referenced by:



0x3b1d06 (BTrees.OIBTree.OITreeSet) is referenced by 0x3b1d01 (BTrees.OOBTree.OOBucket) at level 1

0x3b1d01 (BTrees.OOBTree.OOBucket) is referenced by 0x11c284 (BTrees.OOBTree.OOBTree) at level 2

0x11c284 (BTrees.OOBTree.OOBTree) is _reltoken_name_TO_objtokenset for 0x11c278 (z3c.relationfield.index.RelationCatalog) at level 3

0x11c278 (z3c.relationfield.index.RelationCatalog) is relations for 0x1e (five.localsitemanager.registry.PersistentComponents) at level 4

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x11 (Products.CMFPlone.Portal.PloneSite) at level 5

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x01 (OFS.Application.Application) at level 6

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f6 (persistent.mapping.PersistentMapping) at level 7

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f7 (zope.component.persistentregistry.PersistentAdapterRegistry) at level 8

8< --------------- >8 Stop at root objects

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02f5 (plone.app.redirector.storage.RedirectionStorage) at level 6

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fa (zope.ramcache.ram.RAMCache) at level 7

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fd (plone.contentrules.engine.storage.RuleStorage) at level 8

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x338f13 (plone.app.contentrules.rule.Rule) at level 9

0x338f13 (plone.app.contentrules.rule.Rule) is ++rule++rule-2 for 0x0303 (BTrees.OOBTree.OOBTree) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346961 (plone.app.contentrules.rule.Rule) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b59 (plone.app.contentrules.rule.Rule) at level 11

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b61 (plone.app.contentrules.rule.Rule) at level 12

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fe (plone.app.viewletmanager.storage.ViewletSettingsStorage) at level 9

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x034d (plone.keyring.keyring.Keyring) at level 10

0x034d (plone.keyring.keyring.Keyring) is referenced by 0x02fb (persistent.mapping.PersistentMapping) at level 11

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a32 (plone.keyring.keyring.Keyring) at level 12

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a33 (plone.keyring.keyring.Keyring) at level 13

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x3b3dc4 (pas.plugins.ldap.plonecontrolpanel.cache.CacheSettingsRecordProvider) at level 11

0x3b1d01 (BTrees.OOBTree.OOBucket) is _next for 0x3b1cf1 (BTrees.OOBTree.OOBucket) at level 3

[...]

From this output you can find out that the broken object is (surprise) a item in the RelationCatalog of zc.relation. See the chapter "Frequent Culprits" for information how to deal with these.

Example 1 of using `fsoids.py`

In this and the next example I will use the script fsoids.py to find out where a broken objects actually sits so I can remove or fix it. The easier approach is to use zodbverify but I discuss this approach here since it was your best option until I extended zodbverify and since it might help you to understand the way references work in the ZODB.

$ ./bin/zodbverify -f var/filestorage/Data.fs



INFO:zodbverify:Done! Scanned 120797 records.

Found 116 records that could not be loaded.

Exceptions and how often they happened:

AttributeError: Cannot find dynamic object factory for module plone.dexterity.schema.generated: 20

AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

ModuleNotFoundError: No module named 'Products.Archetypes': 5

ModuleNotFoundError: No module named 'Products.CMFDefault': 20

ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

ModuleNotFoundError: No module named 'fourdigits': 8

ModuleNotFoundError: No module named 'plone.app.controlpanel': 2

ModuleNotFoundError: No module named 'plone.app.stagingbehavior.interfaces'; 'plone.app.stagingbehavior' is not a package: 34

ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

Follow the white rabbit...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907d



oid 0x35907d BTrees.OIBTree.OISet 1 revision

    tid 0x03c425bfb4d8dcaa offset=282340 2017-12-15 10:07:42.386043

        tid user=b'Plone xxx@xxx.de'

        tid description=b'/Plone/it-service/hilfestellungen-anleitungen-faq/outlook/content-checkout'

        new revision BTrees.OIBTree.OISet at 282469

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        referenced by 0x35907b BTrees.OIBTree.OITreeSet at 911946038



[...]

Follow referenced by ...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907b



[...]

referenced by 0x3c5790 BTrees.OOBTree.OOBucket

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3c5790



[...]

referenced by 0x11c284 BTrees.OOBTree.OOBTree

[...]

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x11c284



[...]

referenced by 0x3d0bd6 BTrees.OOBTree.OOBucket

[...]

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3d0bd6



[...]

referenced by 0x11c278 z3c.relationfield.index.RelationCatalog

[...]

Found it!!!!!

Example 2 of using `fsoids.py`

In this example zodbverify found a trace of Products.PloneFormGen even though you think you safely uninstalled the addon (e.g. using https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/addons.py#L11)

Then find out where exists in the tree by following the trail of items that reference it:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x372d00

oid 0x372d00 Products.PloneFormGen.content.thanksPage.FormThanksPage 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision Products.PloneFormGen.content.thanksPage.FormThanksPage at 912841984

        referenced by 0x372f26 BTrees.OIBTree.OIBucket at 912930339

        references 0x372e59 Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5a Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5b OFS.Folder.Folder at 912841984

        references 0x372e5c Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5d Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5e Persistence.mapping.PersistentMapping at 912841984

    tid 0x03d40a3e52a41633 offset=921078960 2019-11-25 17:02:19.368976

        tid user=b'Plone pbauer'

        tid description=b'/Plone/rename_file_ids'

        referenced by 0x2c1b51 BTrees.IOBTree.IOBucket at 921653012

Follow referenced by until you find something...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x2c1b51

oid 0x2c1b51 BTrees.IOBTree.IOBucket 1 revision

    [...]

Here I skip the trail of referenced by until I find 0x280184 five.intid.intid.IntIds:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x280184

oid 0x280184 five.intid.intid.IntIds 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision five.intid.intid.IntIds at 8579054

        references 0x28018c <unknown> at 8579054

        references 0x28018d <unknown> at 8579054

    tid 0x03d3e90c4d3aed55 offset=915868610 2019-11-19 19:24:18.100824

        tid user=b' adminstarzel'

        tid description=b'/Plone/portal_quickinstaller/installProducts'

        referenced by 0x02f6 persistent.mapping.PersistentMapping at 915868690

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x1e five.localsitemanager.registry.PersistentComponents at 915898834

That is the IntId-Catalog from zope.intid. The problem seems to be that similar to the zc.relation catalog rerefences to broken objects stay in the catalog and need to be removed manually.

Here is a example of how to remove all broken objects from the catalog in a pdb-session:

(Pdb++) from zope.intid.interfaces import IIntIds

(Pdb++) from zope.component import getUtility

(Pdb++) intid = getUtility(IIntIds)

(Pdb++) broken_keys = [i for i in intid.ids if 'broken' in repr(i.object)]

(Pdb++) for broken_key in broken_keys: intid.unregister(broken_key)

(Pdb++)

(Pdb++) import transaction

(Pdb++) transaction.commit()

After packing the DB the problem is gone. o/

Other Options

Use zodbbrowser to inspect the ZODB.: It is Zope3 app to navigate a ZODB in a browser. At least I had problems getting it to run with a Plone-ZODB.
Use zc.zodbdgc: This tool can validate distributed databases by starting at their root and traversing to make sure all referenced objects are reachable. Optionally, a database of reference information can be generated.
Use collective.zodbdebug: A great tool to build and inspect reference-maps and backreference-maps of a ZODB. So for it does not work with Python 3 yet. Some if its features are also part of zodbverify.

Frequent Culprits

IntIds and Relations

The IntId-Tool and the relation-catalog are by far the most requent issues, especially if you migrated from Archetypes to Dexterity.

There may be a lot of RelationValues in these Tools that still reference objects that cannot be loadedif these removed objects were not properly removed.

The following code from collective.relationhelpers cleans up the IntId- and Relation-catalog but keeps relations intact. For large sites it may take a while to run because it also needs to recreate linkintegrity-relations.

from collective.relationhelpers.api import cleanup_intids

from collective.relationhelpers.api import purge_relations

from collective.relationhelpers.api import restore_relations

from collective.relationhelpers.api import store_relations



def remove_relations(context=None):

    # store all relations in a annotation on the portal

    store_relations()

    # empty the relation-catalog

    purge_relations()

    # remove all relationvalues and refs to broken objects from intid

    cleanup_intids()

    # recreate all relations from a annotation on the portal

    restore_relations()

For details see https://github.com/collective/collective.relationhelpers/blob/master/src/collective/relationhelpers/api.py

Annotations

Many addons and features in Plone store data in Annotations on the portal or on content.

It's a good idea to check IAnnotations(portal).keys() after a migration for Annotation that you can safely remove.

Here is a example where wicked (the now-removed wiki-style-editing feature of Plone) stored it's settings in a Annotation:

def cleanup_wicked_annotation(context=None):

    ann = IAnnotations(portal)

    if 'plone.app.controlpanel.wicked' in ann:

        del ann['plone.app.controlpanel.wicked']

Another example is files from failed uploads stored by plone.formwidget.namedfile in a annotation:

def cleanup_upload_annotation(context=None):

    # remove traces of aborted uploads

    ann = IAnnotations(portal)

    if ann.get('file_upload_map', None) is not None:

        for uuid in ann['file_upload_map']:

            del ann['file_upload_map'][uuid]

Appendix

Migrating a ZODB from py2 to py3

Since people often encounter issues with their ZODB after migrating here is a quick dive into migrating a ZODB from Python 2 to Python 3.

The migration is basically calling the script zodbupdate in py3 with the parameter --convert-py3.

$ ./bin/zodbupdate --convert-py3

You need to pass it the location of the database, the defaul-encoding (utf8) and a fallback-encoding (latin1) for items where decoding to utf8 fails.

Example:

$ ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8 --encoding-fallback latin1



Updating magic marker for var/filestorage/Data.fs

Ignoring index for /Users/pbauer/workspace/projectx/var/filestorage/Data.fs

Loaded 2 decode rules from AccessControl:decodes

Loaded 12 decode rules from OFS:decodes

Loaded 2 decode rules from Products.PythonScripts:decodes

Loaded 1 decode rules from Products.ZopeVersionControl:decodes

Committing changes (#1).

After that you should be able to use your ZODB in Python 3.

The process in a nutshell:

First, run bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff yet! This will detect and apply several explicit and implicit rename rules.
Then run bin/instance zodbverify. If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate. But you can still choose to migrate to py3 if this shows errors.
Using Python 3 run bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8
For good measure, on Python 3 run bin/instance zodbverify.

Read the docs: https://docs.plone.org/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.html

Dealing with oids

Transforming oids from int to hex and text and vice versa:

>>> from ZODB.utils import p64

>>> oid = 0x2c0ab6

>>> p64(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'



>>> from ZODB.utils import oid_repr

>>> oid = b'\x00\x00\x00\x00\x00,\n\xb6'

>>> oid_repr(oid)

'0x2c0ab6'



>>> from ZODB.utils import repr_to_oid

>>> oid = '0x2c0ab6'

>>> repr_to_oid(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'

Get a path for blobs:

from ZODB.blob import BushyLayout

if isinstance(oid, int):

    # e.g. oid = 0x2c0ab6

    from ZODB.utils import p64

    oid = p64(oid)

return BushyLayout.oid_to_path(None, oid)

Load a obj by oid from ZODB in a pdb-prompt:

oid = 0x2c0ab6

from ZODB.utils import p64

app = self.context.__parent__

obj = app._p_jar.get(p64(oid))

TODO

Finish, merge and document https://github.com/plone/zodbverify/pull/8

Links

Quite a lot of people from the Plone/Zope communities have wriotten about this issue. I learned a lot from these posts:

Automated subtitles – destructuring a successful Plone CMS integration

Posted by Asko Soukka on June 22, 2020 12:00 PM

Never underestimate the importance of being able to make changes to your software – especially when they are critical to your business processes.

Our university has its own audio and video publishing platform, Moniviestin. The first version was released in 2003 – well before Youtube. The latest major iteration was done in 2010, and is built on top of Plone CMS platform, with microservice architecture based video encoding pipeline. After 10 years and counting, we had yet another critical feature request: most of the new published recordings must have subtitles, as automatically as possible.

Once the team had benchmarked the available automatic speech recognition (ASR) services for Finnish speech, they selected Sanelius ASR HTTP API from Ääni Company.

Now they had the specification, but how did relying on Plone help with the integration?

From “Publish” to “Publish with subtitles”

To make the end user experience as convenient as possible, the team decided to connect the automated subtitle generation into the current video publication workflow.

In Plone, every content object may be supported with one or more state managing workflows. For example, our video content pages, called “Media pages”, have one workflow for managing the publication process and another one for managing the video encoding process. Because Plone is designed to work as a web publishing platform out-of-the-box, only the publication workflow is exposed for end-users by default.

The obvious starting point for the integration was to branch the publication workflow with a new path: “Publish with subtitles”. That was enough to provide the required user interface for the feature, just next to the familiar “Publish” action. The new branch in the worklow made also possible the other required states and transitions to support the actual integration.

What can be seen, can be automated

Our platform had support for manually configurable subtitles already. Plone is based on hierarchical object database, not unlike filesystem with folders and files. Therefore, our platform was built to represent its content with folder-like “Media page” containers, which could contain any amount of related attachment items, like slides, lecture notes, and… subtitles!

So, “Subtitles” was already a feature on our CMS based platform, and configurable within “Media page” manually through the user interface. But not only was “Subtitles” available to the end-users, it was also available through Plone REST API. And Plone REST API provided out-of-the-box most of the necessary actions to fetch “Media pages” waiting for subtitles, post new subtitles, update existing subtitles, and confirm the updates according to the workflow.

Having workflow changes, addable “Subtitles”-content and scriptable REST API in place, one more Plone automation feature still needs to be mentioned: configurable event based actions, also known as “content rules”:

For automated subtitles, Plone content rules made it simple to trigger automation service when “Publish with subtitles” was selected. Similarly it made simple to configure email notifications when automated subtitles were received.

Finally, let robots do the hard work

RPA, or Robotic Process Automation, is usually associated with expensive automation platforms with visual programming features. Yet, deep inside, this fancy term could simply mean any kind of script automation, being triggered by external events or timer, to perform some business value providing action. Actually, a very useful approach for casual automation…

For automated subtitles, our team did not need to build tight integration between our video publishing platform and the selected ASR service. Neither did the team need to build any new continuosly running services to handle the integration. Our team simply needed to ensure that both ends had consumable HTTP APIs, and then write required scripts (“robots”) to handle the required communication between services.

The most obvious place for the required integration scripts was our existing Jenkins based RPA platform: Jenkins provides us job configuration, secrets management, webhooks endpoints, scheduled executions and archival of execution logs. Simply everything our team needed to manage the execution of these subtitle automation tasks.

In addition, our RPA Jenkins workers were already powered by Nix package manager to provide all run-time dependencies for the automation scripts. I was told that just using Nix saved up to day in development time, because it provided complete and up-to-date ffmpeg installation for the integration scripts without any additional effort.

Let there be subtitles!

Welcome our full subtitle automation story:

Video author selects “Publish with subtitles”.
Plone content rule calls Jenkins webhook to schedule a new robot.
Robot reads pending task from Plone, streams the video, extracts and converts the audio track, and submits the track to ASR service.
Jenkins schedules a new job to poll pending ASR jobs.
Robot reads completed task from ASR service, downloads the text, converts it into subtitles format and uploads the file to Plone.
Plone content rule notifies the video author with email that automatically generated subtitles are now available.
???

Profit.

Static is fast, but CMS still required – a JAMstack story

Posted by Asko Soukka on June 12, 2020 12:00 PM

An another iteration of our university’s new study guide web site has been completed. The project that started more than a year ago as a JAMstack experiment with GatsbyJS and Hasura, has finally matured enough to get its long expected expansion: a content management system!

In the beginning there was data. A lot of it. A great amount of granular JSON chunks, to be turned into fast and well connected study guide web site. Regularly updated, of course. For years we had solved similar use cases by building and synchronizing CMS content out of more or less structured data. This time we had not enough resources for a such “sophisticated” CMS integration, but we had to look for more agile alternatives – think out of the box. We chose to go JAMstack, with GatsbyJS.

Hasura – the magical GraphQL gateway

At first, of course, we had to make sense of our data. GatsbyJS requires to use GraphQL queries to select the published content. By chance, we found Hasura, which is designed to turn any PostgreSQL database into well-connected GraphQL API. So, we built a pipeline to dump our JSON data into JSONB columns within simple PostgreSQL tables. Once the data was in the database, and we learned which parts we really needed, we could build dynamic database views to expose the data exactly as we wanted.

As designed, Hasura was able to publish those views as GraphQL types and connect them with the relations we needed. Suddenly we had complete GraphQL API for our data. Almost, but not quite entirely unlike, magic. Awesome. Really.

But publishing all that structured data with fast and accessible user experience was not enough. More information was required to be included on the site. This time with images, video embeds and attachments. And the master system was not designed to handle that.

After all, a real CMS was required.

Volto – a breeze of fresh air

Lucky us, we had just right CMS product and experience available. Plone CMS with its latest user interface, Volto, provided us both hierarchical object database and modern user interface required for managing the additional content. Plone shines in managing content in folder tree like hierarchies, and sharing access rights for them accordingly. Volto, on the other hand, makes Plone snappier and easier to use than ever.

And when it comes to Plone as a data source for GatsbyJS: I personally mentored Google Summer of Code students both in 2018 and 2019, and then continued the work, to make sure that Plone integrates perfectly with any GatsbyJS project.

The last piece in our puzzle was to connect Volto authored Plone CMS content with our structured data from the Hasura powered GraphQL API. The flexibility of Plone CMS, with fresh customization possibilities provided by Volto, enabled our solution:

Plone ships with out-of-the-box customizable structured content types. Without any custom code, we were able to enhance our Volto-editable pages with metadata fields to store the connecting information. This also made the data available in Plone REST API for GatsbyJS data source integration.

, 2) Thanks to Volto user interface being customizable with our own ReactJS code, it was possible to customize the select widget of our primary connecting field, to search our Hasura GraphQL API for all the possible value options, to be saved with the content page.

All this required successful teamwork, not only with a few developers doing the technical implemenation, but also with the dozens of our content editors creating and connecting the actual content. That said, we successfully reached our goal:

What you see, edit and connect in Volto…

…is what you get in our GatsbyJS built study guide web site, knitted together with the original study guide data.

Something old, something new and something blue.

The perfect match.

Refreshing CMS, in a theme, with Plone

Posted by Asko Soukka on April 13, 2020 12:00 PM

“How hard can it be? It is just a theme…”

Of course, it was. Unless it was a collection of configurable interactive components. With features like tabbed carousels, photo filters, hyphenation, and syndication of news or calendar feeds from various sources. All responsive. All accessible. All reusable around the site. All with multilingual user interface elements, when required.

Some might confuse that for requirement specification of a new CMS/WCM project. For us it was just a theme refresh for the current installation. And, to be honest, thanks to Plone, the hardest part really was the CSS.

Real-time layouts with Plone Mosaic

Being able to see the content in its themed context while editing it, has always been the definitive part of Plone editor experience. WYSIWYG to the max, they say. There are still options to keep that Plone promise alive in the era of “modern web tech”. Our choice has been Plone Mosaic site layouts.

Plone Mosaic site layouts turn the principles of traditional CMS theming upside down (continuing the tradition of Plone Diazo). Instead of theming content in CMS, the CMS content gets merged into theme, themed Plone Mosaic site layouts.

We build our themed layouts with Webpack. Plone Webpack integration allows us to bring in all the bells and whistles we need from the huge open source JavaScript ecosystem without extra effort. And thanks to Patternslib and Webpack code-splitting, huge libraries like MathJax are only loaded when required.

Eventually, the CMS content gets pulled into Plone Mosaic site layouts with “tiles” and “panels”: Tiles are placeholders for any CMS content from page title to body text. Panels are customizable areas, where more tiles can be placed in customizable grid layouts. And when that is not enough, some things can still be tweaked with Plone Diazo XSLT rules…

Configurable blocks with Theme Fragments

The days when it was enough to theme the existing features of a CMS are long gone. On the contrary, nowadays it seems that themes redefine the required features. Lucky us, not only was Plone there from the very beginning, Plone itself started as a themed user interface for Zope Content Management Framework. While the details have changed, in my opinion, Plone could still market itself as a low-code platform for web content management.

Plone Theme Fragments provide flexible way to enrich theme with configurable functional blocks. Minimally, theme fragments are re-usable static HTML fragments usable around the theme. But they can also use all the power of Plone templating language to render the current content in custom manner. Even more, fragments can be bundled with Python functions to allow complex business logic calling most of the Plone backend API while keeping the templates itself simple.

With Plone Mosaic, theme fragments can be also used as tiles in any Mosaic layout. And when that itself is not flexible enough, theme fragment tiles can be made configurable with the full power of Plone Supermodel XML schemata.

All this. Simply as part of any Plone theme. No wonder we have been using these beasts a lot.

Everything bundled with Theme Site Setup

When implementing a theme refresh for an existing web site cluster with dozen of independent CMS installations with tens of thousands of individual pages, it is important to be able to iterate fast. And for theming a Plone cluster that means, to be able to update the theme without need to restart the backend services after each update.

Plone Theme Site Setup add-on for the rescue! Thanks to Theme Site Setup, our theme packages may include everything we need from the usual theming resources and Theme Fragments, to Plone Mosaic layouts, custom language localization catalogs and Plone site configuration changes (like customizing cached image scales).

In practice, we use Plone Webpack integration to produce complete theme packages with all the required resources supported by Plone Theme Site Setup. Then we use Plone Theme Upload to upload the resulting package to our sites on demand. No restarts needed.

Personally cached with Varnish ESI

At the end, no features matter if the resulting web site is slow or its content is not up-to-date. Unfortunately, these two requirements are often contradicting each other. Especially on web portal front pages that mostly aggregate the current content from all the other pages.

Fortunately, Plone Mosaic was designed with ESI (Edge Side Includes) and tile specific caching configuration in mind. With simple customization of Plone Mosaic rendering pipeline and Plone caching rules, we have been able to achieve everything we wanted:

Different parts of our pages are cached for different periods of time. For example, news listings tiles are invalidated from cache in every few minutes, while the rest of the page is only updated when modified.
Cache is shared with anonymous and logged-in users when safe. For example, the same cached versions of header and footer tiles get shared between all users.
Also most of the tiles for logged-in users can be securely cached: users with the same set of user roles share the same cached version.
Thanks to Varnish’ recursive ESI support, we are able to provided cached personalized news listing tiles: the first response is a non-cached tile rendering just the ESI-reference for the cached version, and all users with matching configuration get the same cached version. Fast.

Finally, Plone Webpack integration allows us to build a theme with all its front-end resources server from a separate server, possibly from a CDN, in an optimized manner. Allowing all our sites with the same theme share the same resources, and let Plone to focus on managing and serving the content.

All that said, and as already been said, thanks to Plone, the hardest part really was (and remains to be) the CSS.

Creating Plone content with Transmogrifier on Python 3

Posted by Asko Soukka on February 25, 2020 12:00 PM

TL;DR; This blog post ends with minimal example of creating Plone 5.2 content with Python 3 compatible Transmogrifier pipeline with command line execution.

Years ago, I forked the famous Plone content migration tool Transmogrifier into a Plone independent and Python 3 compatible version, but never released the fork to avoid maintenance burden. Unfortunately, I was informed that my old examples of using my transmogrifier fork with Plone no longer worked, so I had to review the situation.

The resolution: I found that I had changed some of the built-in reusable blueprints after the post, I updated the old post, fixed a compatibility issue related to updates in Zope Component Architecture dependencies, and tested the results with the latest Plone 5.2 on Python 3.

Transmogrifying RSS into Plone

So, here goes a minimal example for creating Plone 5.2 content with Python 3 Transmogrifier pipeline using my fork:

At first ./buildout.cfg for the Plone instance:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
user = admin:admin

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Then buildout must be run to create the instance with a Plone site:

$ buildout

Next the transmogrifier ./pipeline.cfg must be created to define the pipeline:

[transmogrifier]
pipeline =
    from_rss
    prepare
    create
    patch
    commit

[from_rss]
blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url = http://rss.slashdot.org/Slashdot/slashdot

[prepare]
blueprint = transmogrifier.set
portal_type = string:Document
id = python:None
text = path:item/summary
_container = python:context.get('slashdot') or modules['plone.api'].content.create(container=context, type='Folder', id='slashdot')

[create]
blueprint = transmogrifier.set
modules = plone.api
object = python:modules['plone.api'].content.create(container=item.pop('_container'), type='Document', **item)

[patch]
blueprint = transmogrifier.transform
modules = plone.app.textfield
patch = python:setattr(item['object'], 'text', modules['plone.app.textfield'].value.RichTextValue(item['object'].text, 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the execution of transmogrifier with Plone site as its context (remember that this version of transmogrifier also works outside Plone ecosystem, but for a convenience transmogrify-script also supports calling with instance run):

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

This example should result with the latest Slashdot posts in a Plone site. And, because this example is not perfect, running this again would create duplicates.

Transmogrifying JSON files into Plone

There’s never enough simple tutorials on how to build your own Transmogrifier pipelines from scratch. Especially now, when many old pipeline packages have not been ported to Python 3 yet.

In this example we configure a buildout with local custom Transmogrifier blueprints in python and use them to do minimal import from a JSON export generated using collective.jsonify, which is a one of many legacy ways to generate intermediate export. (That said, it might be good to know, that nowadays trivial migrations could be done with just Plone REST API and a little shell scripting.)

At first, we will define a ./buildout.cfg that expects a local directory ./local to contain a Python module ./local/custom and include ZCML configuration from ./local/custom/configure.zcml:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
    plone.restapi
user = admin:admin
extra-paths = local
zcml = custom

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Before running buildout we ensure a proper local Python module structure with:

$ mkdir -p local/custom
$ touch local/custom/__init__.py
$ echo '<configure xmlns="http://namespaces.zope.org/zope" />' > local/custom/__init__.py

Only then we run buildout as usually:

$ buildout

Now, let’s populate our custom module with a Python module ./local/custom/blueprints.py defining a couple of custom blueprints:

# -*- coding: utf-8 -*-
from transmogrifier.blueprints import Blueprint

import json
import pathlib


class Glob(Blueprint):
    """Produce JSON items from files matching globbing from option `glob`."""
    def __iter__(self):
        for item in self.previous:
            yield item
        for p in pathlib.Path(".").glob(self.options["glob"]):
            with open(p, encoding="utf-8") as fp:
                yield json.load(fp)


class Folders(Blueprint):
    """Minimal Folder item producer to ensure that items have containers."""
    def __iter__(self):
        context = self.transmogrifier.context
        for item in self.previous:
            parts = (item.get('_path') or '').strip('/').split('/')[:-1]
            path = ''
            for part in parts:
                path += '/' + part
                try:
                    context.restrictedTraverse(path)
                except KeyError:
                    yield {
                        "_path": path,
                        "_type": "Folder",
                        "id": part
                    }
            yield item

And complete ZCML configuration at ./local/custom/configure.zcml with matching blueprint registrations:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">

  <include package="transmogrifier" file="meta.zcml" />

  <transmogrifier:blueprint
      component=".blueprints.Glob"
      name="custom.glob"
      />

  <transmogrifier:blueprint
      component=".blueprints.Folders"
      name="custom.folders"
      />

</configure>

Now, by using these two new blueprints and minimal content creating pipeline parts based on built-in expression blueprints, it is possible to:

generate new pipeline items from exported JSON files
inject folder items into pipeline to ensure that containers are created before items (because we cannot quarentee any order from the export)
create minimal Folder and Document objects with plone.api.

[transmogrifier]
pipeline =
    generate_from_json
    generate_containers
    set_container
    create_folder
    create_document
    commit

[generate_from_json]
blueprint = custom.glob
glob = data/**/*.json

[generate_containers]
blueprint = custom.folders

[set_container]
blueprint = transmogrifier.set
_container = python:context.restrictedTraverse(item["_path"].rsplit("/", 1)[0])

[create_folder]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Folder"
modules = plone.api
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Folder", id=item["id"])

[create_document]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Document"
modules =
  plone.api
  plone.app.textfield
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Document", id=item["id"], title=item["title"], text=modules['plone.app.textfield'].value.RichTextValue(item["text"], 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the pipeline can be run and content imported with:

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

Obviously, in a real migration, the pipeline parts [create_folder] and [create_document] should be implemented in Python to properly populate all metadata fields, handle possible exceptions, etc, but consider that as homework.

If this post raised more questions than gave answers, please, feel free to ask more at: https://github.com/collective/transmogrifier/issues.

Why Upgrade?

Posted by Jazkarta Blog on February 21, 2020 03:52 PM

Plone 5.2, The Future-Proofing Release: Python 3 and REST API

Technology never stands still.

It’s tempting to think of technology investments as discrete expenditures that permanently solve a problem, but that would be a mistake. A new website that costs $25K, $50K, $100K or more feels like it should last forever. But technology ages and an organization’s needs evolve. Everyone is happy for a short while after the website is completed, but then they become less and less happy as it works less and less well.

A better approach is to not think of technology needs as being solved by big, herculean efforts that happen occasionally, but as an ongoing program that requires ongoing resources. This is what the University of Minnesota Press has done. Since their current website’s initial launch in 2011, they have:

2012: Added a searchable bibliography to the Test Division portion of the website
2015: Done a responsive redesign so that the website works seamlessly on mobile devices
2018: Upgraded the website’s e-commerce infrastructure with modern components providing improved PCI compliance
Plus they’ve had a yearly support contract to fix bugs, add features, and keep up with minor version upgrades

This pattern of ongoing investment is typical of our clients. And because of technology changes that have occurred over the last few years, a new round of investments has become imperative: upgrades.

Since 2011 the Press website has been running on version 4 of the content management system Plone, and version 2 of Python, the programming language used to implement Plone. Those versions are nearing obsolescence.

Plone 5 has been out since 2016, and Plone 6 is expected soon. When Plone 6 is released, the Plone security team will end official support for Plone 4.
Python 3, a major, backwards-incompatible release with many new features, has been out since 2008. Official support for Python 2 ended January 1, 2020.

Because of this, it became essential for the Press – like other organizations that use Plone – to budget for major version upgrades of its website technology stack. This long-term investment will ensure that all technology components are stable, supported, and up to date.

The Plone 5 version upgrade will also provide benefits to users, editors, and website developers.

Users will see:

Pages that render 15-20% faster due to a new templating engine
Improved accessibility compliance

Editors and admins will see a number of new features under the hood:

An improved editing toolbar
The latest version of the WYSIWYG rich text editor (TinyMCE)
Facebook OpenGraph meta tags and Twitter card support
Bulk editing operations such as adding multiple files and images at once
The ability to quickly find, sort, reorder, and select content items on the contents view
Automatic CSRF (cross-site request forgery) protection integrated into the database layer

Website developers will be able to use:

All the features in the latest Python
Plone’s improved and easier to use content type framework, Dexterity, as well as other new features in the code

Visit Plone.com to read more about the advantages of Plone 5.

The good news is that this upgrade work can be done in phases – meaning the work can be budgeted over several years if necessary.

A Plone 5.1 upgrade, which includes migrating Plone’s core content types to Dexterity.
Migrating custom content types to Dexterity and replacing any add-ons that are not compatible with Plone 5.2.
A Plone 5.2 and Python 3 upgrade.

Phases 1 and 2 must be done before phase 3 because the old content type framework (Archetypes) is not supported in Python 3.

The end result of this upgrade path is to open up a world of possibilities to organizations using Plone. Out of the box Plone 5.2 includes:

plone.restapi, which supports the full set of Plone features (users, groups, roles, workflow, navigation, search, even breadcrumbs)
Volto, a modern Javascript front end for Plone based on React

These new components are game changers. In particular, the REST API allows Plone to integrate easily with other systems and to operate as a headless CMS – with the content delivery front end decoupled from the back end.

Now that’s worth upgrading for.

A Volto gotcha when dealing with async calls

Posted by PloneExpanse on December 11, 2019 08:35 PM

Just some quick notes, in case this might help someone. After quite a bit of time and tests in trying to use asyncConnect to get data in a Volto component view (strictly focusing on the SSR side), I’ve realized that what I’m trying to do is not supported by the redux-connect library. In Volto, right now there are two components that use asyncConnect: App.jsx and Search.jsx. The purpose of asyncConnect is to have the server side rendered page “dynamic”, depending on the input from the originating request.

Speedup volto razzle builds

Posted by PloneExpanse on November 17, 2019 12:58 PM

I’ve been looking for a way to speedup Volto razzle/webpack builds, both while developing and for “production” mode, when building the final bundle. Fortunately, this solution exists and it’s extremely easy to integrate. Let’s define the problem, to see how to approach it: what is Volto actually? What do you get when you open, in your browser, a Volto frontend Plone website? To greatly simplify (and I hope I didn’t get anything wrong as I am not a Volto core developer):