This post is a summary of the usability testing done as a joint collaboration between design contributors and FSE Outreach program participants. For starters, big thank you to everyone who contributed both as designers and participants. Having watched each of the following videos, I’m really struck by how seriously everyone took this effort and how much kindness existed in each interaction, even when sharing tough feedback or stumbling over the experience. This was an experiment and it was a success because of how each of you engaged.

Before digging into the videos themselves, I want to note that I had to scale down many, if not all, of the videos in order to get them to upload properly to Make Test. For full access to the highest quality recordings, you can view them in this Google Drive folder that was set up for this series of tests. Separately, the intent was to have an equal split between both flows but that wasn’t always followed for each participant.

Finally, everyone who participated, whether as a design contributor or participant, will get a Test Contributor badge added to their profile as soon as possible if you don’t already have one!

• Thank you to our design facilitators: @javiarce @critterverse @estelaris @ericcarco and @colourpress.
• Thank you to our participants: @piermario @femkreations @contentiskey @samanthaxmunoz @mroehricht @arturgrabo @krupaly2k @oglekler @chzumbrunnen @beckej @hhoskins Matthias Reinholz (in process of getting username).

Overall takeaways

High level summary

Across each experience, the following commonalities stood out:

• Lack of awareness around List View and, even for those who do use it, frustration around dragging and dropping items.
• Desire for more visual indicators of where content would be added, whether when adding patterns or trying to create a new template part without it being baked inside another one. This caused repeated placing of patterns within patterns or template parts within template parts.
• Missing knowledge around patterns, including discovering them to begin with or finding how they are sorted by categoryCategory The 'category' taxonomy lets you group posts / content together that share a common bond. Categories are pre-defined and broad ranging..
• Great difficulty using the navigation blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience., whether to do something more advanced like create a different experience on mobile or as simple as understanding how to add items to the menu based on the current placeholder state.
• Easier to replace a template part while creating from scratch proved painful, even if folks went into the isolated template part mode. This is due to things like confusion around building simple layouts, details around setting the correct template part area, the impact of duplicating template parts, and restrictions in what can be done to edit a template part in isolated mode.
• Clear confusion around the various sidebars, where to find settings, what impacts a site globally (like Styles), etc.

Bugs

• When duplicating a Navigation or Template Part block, it’s not clear why changing one changes the other.
• Block Toolbar Regression – Toolbar obstructs selected blocks.
• Navigation Block: prevent submenu items from merging.
• Navigation toolbar overlaps on submenu links when we open sidebar settings or when used on smaller viewport.

Feature requests

• [New] Ability to find patterns when searching “pattern” in the Quick Inserter.
• More robust autosaving.
• Want to be able to create new post/page from within site editor (no direct issue to link to as this is part of broader Site Editor efforts).
• Clearer font size labeling.
• Consistent dimension controls.
• Desire to control the styling of interactive states.
• Better organization for block transforms to include template parts.
• Template editing: convert semantic patterns to template parts upon insertion.
• Make it possible to edit template part properties in focus mode.
• Fixed Position Header and Footer Template Parts.
• Show Template Part color settings when in isolated mode (closed).

Template Part Testing

Level listed under each video reflects a self reported level of comfort with FSE with 1 being not all familiar and 10 reflecting expertise. For any listed as N/A, the participant was not asked.

Video 1

• Level: 2-3
• Didn’t use list view.
• Replaced the template part but wasn’t able to create a new one from scratch.
• Navigation block was identified as a major pain.
• Went back to the overall template list to apply the template part to another template but it felt like the wrong place even though it was the right spot to go to (around 7 minutes in).
• Unable to complete the final task of applying a template part to another template.

“It’s definitely different than the WordPress I’m used to. I’m more used to operating through the dashboard than a visual editor which I think is why I was unsure of what I was doing. Otherwise, as far as changing the headerHeader The header of your site is typically the first thing people will experience. The masthead or header art located across the top of your page is part of the look and feel of your website. It can influence a visitor’s opinion about your content and you/ your organization’s brand. It may also look different on different screen sizes., was absolutely quick and painless. Once I learned what some of the other options did it was an easy way to align it how I wanted it to.”

Video 2

• Level: 6 or 7.
• Didn’t use list view.
• Created a template part within the editor to start.
• Created a template part within a template part within a template part, causing the final template part to be smushed within the editor.
• Left to go to template part focus mode to try to add navigation since the + button for the new template part looked so different than what he was used to (5:10-6:00).
• The navigation block placeholder felt unusable and it wasn’t clear how best to add items to it with the + button being so far removed from the experience.
• He tried to use “Manage menus” thinking that would allow for a way to add items only to reach a dead end.

Video 3

• Level: 7 or 8
• Didn’t use list view, except with the navigation block’s list view.
• Went to template parts > add new to add a new template part.
• Couldn’t edit background or basic colors in isolated template part mode. Instead he used Styles which changed the whole site, not just the template part without realizing it would do that (around 6 minutes). This touches on a closed issue.
• Isolated template part mode has an easier flowFlow Flow is the path of screens and interactions taken to accomplish a task. It’s an experience vector. Flow is also a feeling. It’s being unselfconscious and in the zone. Flow is what happens when difficulties are removed and you are freed to pursue an activity without forming intentions. You just do it.

  Flow is the actual user experience, in many ways. If you like, you can think of flow as a really comprehensive set of user stories. When you think about user flow, you’re thinking about exactly how a user will perform the tasks allowed by your product.Flow and Context

  to see bounds of a template part, assign an area, etc but then has huge limitations in terms of customization.
• Wants the ability to create a sticky header and a way to add all new pages to the navigation block.

“It’s a little bit confusing, what you’re really changing and where it works…It’s a lot of understanding that’s needed. I think for an ordinary user it’s very difficult.”

Video 4

• Level: N/A
• Used List View.
• Went back to template parts immediately to create a new one and noted that he usually creates template parts with code.
• At first, duplicated the template part, which runs into this issue. Later, he made a template part by converting a group block using list view. 
• He found the header within a header for Twenty Twenty-Two confusing to navigate.
• He shared that he uses a hack where he adds two navigation blocks: one with mobile always on and one with it always off. From there, will mobile always on, uses custom CSSCSS CSS is an acronym for cascading style sheets. This is what controls the design or look and feel of a site. so it only displays on small screen sizes. 
• New header template part was only available when adding a generic template part and not in the Replace modal. This is because when he originally set up the template part (6:54), he selected general rather than header. He couldn’t find a way to assign the right area for the template part (change from General to Header) within the isolated template part mode.

“This is something that is really annoying to me in the full site editor… moving blocks into different groups…I find it very hard to get to the correct hierarchical level…it’s very flimsy and hard for me to get to the right place.”  

“The navigation block for me is really weird to me. If I click on Manage Menus, it’s empty. There’s nothing to do. I really don’t understand it.” 

Video 5

• Level: N/A
• Used List View.
• Went back to the list of template parts to create a new template part and correctly assigned the area.
• Ran into this issue where the navigation toolbar overlaps with the experience of adding a link, making it nearly impossible to figure out what you’re doing.

“The more helper things you have open (ie: sidebars) I think sometimes it could be hard, especially with full site editing. So maybe you need to use a big screen or remember to close out some of the sidebars”.

“Being able to go through the block navigation (list view) is where I can usually find out if I put something in a container that didn’t belong in that container”.

Video 6

• Level: N/A
• Used List View.
• Went back to template parts and then edited a current header, rather than creating a new one.
• Converting page list items in the navigation was confusing in order to edit each link.
• Ran into the experience of menu items merging when deleting, which has been reported before and was noted as intended functionality.
• When adding a template part into template part focus mode, a template part was not created, matching this currently open issue. In theory, this should create a template part within a template part.

Video 7

• Level: 8
• Used List View.
• Used replace to switch out template part.
• Use of group within a template part feels really odd to him. Feels like it’s only there to help control the layout but feels unnecessary in general. 
• Struggled to drop something into a cover block that had a row block inside of it. It wasn’t clear how to add something else to the row block.
• Finds it limiting which blocks are able to be added to the navigation block. Wants mega menu options and ways to display things based on screen size. 
• Expressed a desire for a middle ground between solely relying on what’s defined by a theme vs getting into the nitty gritty details with Styles.

“I have trouble a lot of times with the list view and getting blocks precisely where I want them to be.”

From around 3:25 in the video

“I don’t make my own headers. Just because I found it completely frustrating and I rely on templates to make all my headers that come with the themes because I’ve found I just can’t manipulate it enough no matter how much time I spend on it. I can’t make things that look better than the pre-made templates so I only do small changes from the pre-made templates… Headers are incredibly challenging. I just don’t know how to do it.”

Video 8

• Level: 1 or 2
• Didn’t use List View. 
• Icons for theme blocks feel very confusing and don’t feel similar to other software he uses. 
• Used replace to switch template part out but, because there is a header within a header for Twenty Twenty-Two, this was a bit confusing.
• He wanted to access different header options using the transform menu. Noted the feedback in a related issue.
• Wanted to duplicate the header at first to have a base to start with which would have run into this problem with duplicated headers.
• Eventually, went into the isolated template part mode where he found the slash inserter to be “very tech-y”.
• Ran into an issue in isolated mode where the toolbar blocks the current blocks.
• Struggled to understand that he needed to select the overall navigation block to customize it in the way he wanted as he expected to find customization options within individual navigation links.
• When asked about saving, he expressed that he expected autosave to work/be happening.

“As a kind of a first time user of the interface, I found this very difficult. It doesn’t remind me of anything I’m used to with other applications I’m using…it’s just not easy for a novice…This was a bit of a learning curve just to get to a header…I was quite uncertain on my choices. I wasn’t sure if I was doing the right thing.”

Pattern Testing

Level listed under each video reflects a self reported level of comfort with FSE with 1 being not all familiar and 10 reflecting expertise. For any listed as N/A, the participant was not asked.

Video 1

• Level: 6-7
• Didn’t use List View. 
• When asked to add a pattern with columns, just added the columns block itself.
• She didn’t know that you could filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. by columns for patterns and remarked that “it’s not obvious they are columns” when glancing at patterns.
• Only used patterns from the Inserter.
• Found the layout settings to be very confusing and px dropdown to be very small/hard to see.
• Wanted to be able to create a new post/page from within site editor rather than needing to go to the post editor.

Video 2

• Level: 8
• Used List View.
• Repeatedly felt that visual cues were missing, especially when wanting to see what was a whole pattern rather than a piece of one.
• Didn’t use drag and drop as it’s not clear where a pattern might go.
• Added a pattern by accident into the query loopLoop The Loop is PHP code used by WordPress to display posts. Using The Loop, WordPress processes each post to be displayed on the current page, and formats it according to how it matches specified criteria within The Loop tags. Any HTML or PHP code in the Loop will be processed on each post. https://codex.wordpress.org/The_Loop. causing it to appear throughout the loop rather than outside of it.
• Noted that the font size labeling felt very confusing with the different numbers. There’s an effort underway to improve this experience with “t-shirt” sizing.
• Wants consistent dimension controls.

“My main problem with the editor is that I don’t always get visual cues on where I can add blocks or patterns.”

Video 3

• Level: 6
• Didn’t use List View until prompted.
• When asked to add a pattern with columns, proceeded to add a Columns block rather than adding a pattern. Later, it was clear that this was due to a lack of awareness around patterns in general.
• Tried repeatedly to find patterns in the quick inserter by searching “patterns” leading to this new issue.
• Participant noted she has tried to make at least 10 WordPress sites and regularly struggles with getting the layout to look correct across different screen sizes. 
• In List view, expected to find mover controls rather than relying on drag and drop and didn’t understand why they weren’t there.
• The browser/site randomly crashed (unclear why).
• Noted that she wanted autosave.

Video 4

• Level: N/A
• Used List View and noted that she “always has it open”.
• Once pattern was added into the site editor it was hard to reposition it where she wanted it to go.
• Wanted a way to change hover colors for buttons.
• Learned about patterns from Nick Diego’s SLS about patterns. Otherwise, did not know they exist and doesn’t feel anything clarifies that they are there.

“Having to go to the + sign and then browse all and that’s the only way I know how to get to patterns is super cumbersome” 

Video 5

• Level: 5
• Used List View. 
• Drag and drop totally didn’t work as expected for adding a pattern. Tried to place it within a query loop and the pattern ended up below it in an unexpected way. Realized later that it was a query loop and it made sense that the pattern was placed below but it proved confusing at the time.
• Added a query loop pattern to the template and was expecting from a user’s point of view to be able to edit the text. The experience felt broken when he wasn’t able to until he realized it was a query loop which he was familiar with. 
• Lack of feedback in being unable to edit the query loop content was jarring. Kept wanting to edit the content itself. 
• Felt that the block settings are very hidden for deeper customization and that it’s hard to know how to get to those options when looking at just the editor itself. 
• Added another pattern as an example that ended up chunkily within another pattern.

“I was thinking I could drop it right above this text…when I grabbed it I saw the hand icon so I knew it was draggable. I didn’t have the clearest ever – I didn’t see like a line showing me it’ll drop here. I didn’t think it would work… it dropped below what I was expecting.” 

“I still am not totally sure if I went and looked at the page on the front end what would be there”.

Noted at the very end of the test.

Next Steps

Going forward, I’ll look to facilitate this kind of usability testing experience again, especially since this time I had to turn folks away.

#fse-usability-testing, #fse-usabilty-summary, #usability-testing