planet

Configuration split: deploy subsets of configuration in Drupal 8

Written on November 28, 2016 - 11:01

From the Configuration split project page: "The Drupal 8 configuration management works best when importing and exporting the whole set of the sites configuration. However, sometimes developers like to opt out of the robustness of CMI and have a super-set of configuration active on their development machine and deploy only a subset. The canonical example for this is to have the devel module installed or having a few block placements or views in the development environment and then not export them into the set of configuration to be deployed, yet still being able to share the development configuration with colleagues."

This small utility module for Drupal 8 allows you to exactly achieve this use case (and many others). I will cover two common scenarios, explaining how to configure your site and which commands you need to run:

  1. Have (development/ui) modules installed on your dev environment, uninstalled on production
  2. Have modules installed on your production environment, but not on development

I've also recorded a screencast in which I configure and run all the commands explained underneath. The best way in the end is to play around with the module itself of course.

Configuration split 101

Configuration split exposes a configuration entity which controls what you want to split off. Currently you can

  • blacklist modules: any configuration that this module owns will automatically be blacklisted too.
  • blacklist configuration: settings or configuration entities. These will be removed from the active sync directory.
  • graylist configuration: settings or configuration entities. These will not be removed if they are in the active sync directory, but also not exported if they are not there yet. I won't cover this functionality in this article post, that will be for another time.

After you configured one or more configurations split entities, you can use the drush commands to export and import configuration, based on one or more of those config entities. When it comes to exporting configuration, you can not use the existing drush core command (config-export / cex). Using drush config-import (cim) or the UI to import may still be used, but this depends on your setup. Drupal Console commands are under revision. Important: since feb 2017, due to https://www.drupal.org/node/2855319 and https://github.com/drush-ops/drush/pull/2471, you can use the standard drush commands without any problems!

Each config split entity defines the directory in which the splitted configuration will live in case there are some. Not all modules define configuration, so there's a possibility that this directory is empty after you do an export.

An important technical aspect is that the module does not interfere with the active configuration but instead filters on the import/export pipeline. What simple happens is this: just before the actual writing, it will check the configuration and remove any entries that you don't want to be there. Then your active configuration is written away. Config that doesn't belong into the active configuration will be moved to separate directory. On import, the opposite happens: it will merge settings back in and set modules to the installed state just before core configuration then starts importing.

Last, but not least: you can swap the config.storage.sync service so that the synchronize screen will use the config split config entities for importing. More information is in the README file and in the video.

Scenario 1: modules installed on dev, not on production

Step 1

After you installed the module, go to 'admin/config/development/configuration/config-split' and click on 'Add configuration split'. Naming is important, so we'll enter 'Dev split' as the name for this configuration. We'll also create a directory called 'sync-dev-split'. Now toggle the modules that you don't want to have installed on production. In this case, we're going to blacklist Devel, Devel Kint, Field UI, Views UI and database logging. Additionally, also toggle system.menu.devel. This configuration is owned by the system module, so there's no dependency on the devel module. There's no problem having this config on your production site though, so it's not required to blacklist it.

Optionally, you can also blacklist the configuration split module because it doesn't necessarily have to be installed on production. We're not doing that here, because in the second scenario, we're building further on this one and we want to have it installed on the production environment as well.

Step 2

Go to your command line interface and run following command:

swentel@dev:/home/drupal/drupal-core$ drush csex --split=dev_split

The split option is not required, but when starting to work with the module, it helps you to know which config split will be used for this command. It's possible to override the status of any config split config entity so that eventually, you can omit the split option. Sadly enough, you don't get any feedback (yet), but after the command has run you should see various files in your sync-dev-split directory:

swentel@dev:/home/drupal/drupal-core$ ls sync-dev-split/
dblog.settings.yml  devel.settings.yml  field_ui.settings.yml  system.menu.devel.yml

Step 3

You can now commit your active sync and go to production. You don't necessarily have to put the sync-dev-live directory in your version control because production doesn't need to know about these files at all. Once you have pulled on production, go to the synchronize screen to verify what is staged. You will see that the setting files will be removed and core.extension will be changed uninstalling the development modules and installing configuration split. Since we only have one config split configuration, you can hit 'Import all' here or run drush config-import, (which is the drush core command). Note that if you don't use the UI, you can blacklist the configuration manager module as well.

You might wonder why we don't use the 'config-split-import' command from the module itself: this would import the active sync directory and also include the modules and settings again that we have blacklisted. And that's not what we want. This seems confusing at first, but ultimately, if this is your setup, you just keep on using the core / drush commands to import your staged configuration on production.

# This command can be used now, but not anymore further on when we will add scenario 2.
swentel@live:/home/drupal/drupal-core$ drush cim

Scenario 2: modules installed on production, not on dev

Step 1

This scenario builds further on the first one. Config split is now installed on our live website. Create a new config split configuration called 'Live split'. You will need a new directory for this second config split, so make sure it's there. On production we still want to log events and for that we're going to use the syslog module. Install the module, so that we can now blacklist it for the live split configuration.

Step 2

Go to your command line interface and run following commands:

swentel@live:/home/drupal/drupal-core$ drush csex --split=live_split

Again, no feedback here, but after the command has run you should see the syslog settings file in your sync-live-split directory:

swentel@live:/home/drupal/drupal-core$ ls sync-live-split/
syslog.settings.yml

Doing exports and imports on dev or live

From now on, if you want to import new settings whether it's on your dev or live environment, you can not use the drush core commands anymore. Use following commands:

# to export on your dev environment
swentel@dev:/home/drupal/drupal-core$ drush csex --split=dev_split
# to import on your dev environment
swentel@dev:/home/drupal/drupal-core$ drush csim --split=dev_split

# to export on your live environment
swentel@live:/home/drupal/drupal-core$ drush csex --split=live_split
# to import on your live environment
swentel@live:/home/drupal/drupal-core$ drush csim --split=live_split

The future

Please join us in the issue queue because there's still some work:

  • Optimize the user interface and allow wildcards
  • Add confirmation and feedback in the drush commands

Ultimately, this functionality should live in core. A core issue to discuss this is at https://www.drupal.org/node/2830300.

Topics 

drupal, planet, configuration management

Stop telling users that another user has modified the content in Drupal 8

Written on August 5, 2016 - 09:55

Every Drupal developer knows the following error message (maybe some by heart): The content has been modified by another user, changes cannot be saved. In Drupal 8 the message is even a bit longer: The content has either been modified by another user, or you have already submitted modifications. As a result, your changes cannot be saved. While this inbuilt mechanism is very useful to preserve data integrity, the only way to get rid of the message is to reload the form and then redo the changes you want to make. This can be (or should I say 'is') very frustrating for users, especially when they have no idea why this is happening. In an environment where multiple users modify the same content, there are solutions like the Content locking module to get overcome this nagging problem. But what if your content changes a lot by backend calls ?

On a big project I'm currently working on, Musescore.com (D6 to D8), members can upload their scores to the website. On save, the file is send to Amazon where it will be processed so you can play and listen to the music in your browser. Depending on the length of a score, the processing might take a couple of minutes before it's available. In the meantime, you can edit the score because the user might want to update the title, body content, or add some new tags. While the edit form is open, the backend might be pinging back to our application notifying the score is now ready for playing and will update field values, thus saving the node. At this very moment, the changed time has been updated to the future, so when the user wants to save new values, Drupal will complain. This is just a simple example, in reality, the backend workers might be pinging a couple of times back on several occasions doing various operations and updating field values. And ironically, the user doesn't even have any permission to update one or more of these properties on the form itself. If you have ever uploaded a video to YouTube, you know that while your video is processing you can happily update your content and tags without any problem at all. That's what we want here too.

In Drupal 8, validating an entity is now decoupled from form validation. More information can be found on the Entity Validation API handbook and how they integrate with Symfony. Now, the validation plugin responsible for that message lives in EntityChangedConstraint and EntityChangedConstraintValidator. Since they are plugins, we can easily swap out the class and depending on our needs only add the violation when we really want to. What we also want is to preserve values of fields that might have been updated by a previous operation, in our case a backend call pinging back to tell us that the score is now ready for playing. Are you ready ? Here goes!

Step 1. Swap the class

All plugin managers in Core (any plugin manager should do that!) allow you to alter the definitions, so let's change the class to our own custom class.

<?php
/**
* Implements hook_validation_constraint_alter().
*/
function project_validation_constraint_alter(array &$definitions) {
  if (isset($definitions['EntityChanged'])) {
    $definitions['EntityChanged']['class'] = 'Drupal\project\Plugin\Validation\Constraint\CustomEntityChangedConstraint';
  }
}
?>

For the actual class itself, you can copy the original one, but without the annotation. The constraint plugin manager doesn't need to know about an additional new one (unless you want it to of course).

<?php
namespace Drupal\project\Plugin\Validation\Constraint;

use Symfony\Component\Validator\Constraint;

/**
* Custom implementation of the validation constraint for the entity changed timestamp.
*/
class CustomEntityChangedConstraint extends Constraint {
  public $message = 'The content has either been modified by another user, or you have already submitted modifications. As a result, your changes cannot be saved. In case you still see this, then you are really unlucky this time!';
}
?>

Step 2: alter the node form

We want to be able to know that a validation of an entity is happening when an actual form is submitted. For this, we're adding a hidden field which stores a token based on the node id which we can then use later.

<?php
/**
* Implements hook_form_BASE_FORM_ID_alter() for \Drupal\node\NodeForm.
*/
function project_form_node_form_alter(&$form, &$form_state) {
  /** @var \Drupal\Node\NodeInterface $node */
  $node = $form_state->getFormObject()->getEntity();
  if (!$node->isNew() && $node->bundle() == 'your_bundle' && $node->getOwnerId() == \Drupal::currentUser()->id()) {
    $form['web_submission'] = [
      '#type' => 'hidden',
      '#value' => \Drupal::csrfToken()->get($node->id()),
    ];
  }
}
?>

Step 3: Validating the entity and storing an id for later

We're getting to the tricky part. Not adding a violation is easy, but the entity that comes inside the constraint can't be changed. The reason is that ContentEntityForm rebuilts the entity when it comes in the submission phase, which means that if you would make any changes to the entity during validation, they would be lost. And it's a good idea anyway as other constraints might add violations which are necessary. To come around that, our constraint, in case the changed time is in the past, will verify if there is a valid token and call a function to store the id of the node in a static variable which can be picked up later.

<?php
namespace Drupal\project\Plugin\Validation\Constraint;

use Symfony\Component\Validator\Constraint;
use Symfony\Component\Validator\ConstraintValidator;

/**
* Validates the EntityChanged constraint.
*/
class CustomEntityChangedConstraintValidator extends ConstraintValidator {

  /**
   * {@inheritdoc}
   */
  public function validate($entity, Constraint $constraint) {
    if (isset($entity)) {
      /** @var \Drupal\Core\Entity\EntityInterface $entity */
      if (!$entity->isNew()) {
        $saved_entity = \Drupal::entityManager()->getStorage($entity->getEntityTypeId())->loadUnchanged($entity->id());
        // A change to any other translation must add a violation to the current
        // translation because there might be untranslatable shared fields.
        if ($saved_entity && $saved_entity->getChangedTimeAcrossTranslations() > $entity->getChangedTimeAcrossTranslations()) {
          $add_violation = TRUE;
          if ($entity->getEntityTypeId() == 'node' && $entity->bundle() == 'your_bundle' &&
            $this->isValidWebsubmission($entity->id())) {
            $add_violation = FALSE;

            // Store this id.
            project_preserve_values_from_original_entity($entity->id(), TRUE);
          }

          // Add the violation if necessary.
          if ($add_violation) {
            $this->context->addViolation($constraint->message);
          }
        }
      }
    }
  }

  /**
   * Validate the web submission.
   *
   * @param $value
   *   The value.
   *
   * @see project_form_node_form_alter().
   *
   * @return bool
   */
  public function isValidWebsubmission($value) {
    if (!empty(\Drupal::request()->get('web_submission'))) {
      return \Drupal::csrfToken()->validate(\Drupal::request()->get('web_submission'), $value);
    }

    return FALSE;
  }

}

/**
* Function which holds a static array with ids of entities which need to
* preserve values from the original entity.
*
* @param $id
*   The entity id.
* @param bool $set
*   Whether to store the id or not.
*
* @return bool
*   TRUE if id is set in the $ids array or not.
*/
function project_preserve_values_from_original_entity($id, $set = FALSE) {
  static $ids = [];

  if ($set && !isset($ids[$id])) {
    $ids[$id] = TRUE;
  }

  return isset($ids[$id]) ? TRUE : FALSE;
}
?>

Step 4: copy over values from the original entity

So we now passed validation, even if the submitted changed timestamp is in the past of the last saved version of this node. Now we need to copy over values that might have been changed by another process that we want to preserve. In hook_node_presave() we can call project_preserve_values_from_original_entity() to ask if this entity is eligible for this operation. If so, we can just do our thing and happily copy those values, while keeping the fields that the user has changed in tact.

<?php
/**
* Implements hook_ENTITY_TYPE_presave().
*/
function project_node_presave(NodeInterface $node) {
  if (!$node->isNew() && isset($node->original) && $node->bundle() == 'your_bundle' && project_preserve_values_from_original_entity($node->id())) {
    $node->set('your_field', $node->original->get('your_field')->value);
    // Do many more copies here.
  }
}
?>

A happy user!

Not only the user is happy: backends can update whenever they want and customer support does not have to explain anymore where this annoying user facing message is coming from.

Topics 

drupal, planet, usability

Taking a (Drupal 8) website offline using AppCache

Written on April 11, 2016 - 11:41

A native mobile application which can cache the data locally is a way to make content available offline. However, not everyone has the time and/or money to create a dedicated app, and frankly, it's not always an additional asset. What if browsers could work without network connection but still serve content: Application Cache and/or Service Workers to the rescue!

For Frontend United 2016, Mathieu and I experimented to see how far we could take AppCache and make the sessions, speakers and some additional content available offline using data from within the Drupal site. There are a couple of pitfalls when implementing this, of which some are nasty (see the list apart link at the bottom for more information). Comes in Drupal which adds another layer of complexity, with its dynamic nature of content and themes. Javascript and css aggregation is also extremely tricky to get right. So after trial and error and a lot of reading, we came up with the following concept:

  1. Only add the manifest attribute to all "offline" pages which are completely separate from "online pages", even though they might serve the same content. In other words, you create a sandboxed version of some content of your site which can live on its own. Another technique is a hidden iframe which loads a page which contains the html tag with the manifest attribute. You can embed this iframe on any page you like. This gives you the option to create a page where you link to as an opt-in to get a site offline. Both techniques give us full control and no side affects so that when network is available the site works normally.
  2. You define the pages which you want to store in the cache. They are served by Drupal, but on a different route than the original (e.g. node/1 becomes offline/node/1) and use different templates. These are twig templates so you can override the defaults to your own needs. Other information like stylesheet and javascript files can be configured too to be included.
  3. The manifest thus contains everything that we need to create the offline version when your device has no network connection. In our case, it contains the list of speakers and sessions, content pages and some assets like javascript, stylesheet, logo and images.

Note: currently, the website isn't app-cache enabled anymore, preparing for 2017.

Offline in the browser or on the homescreen

Go to the Offline homepage of Frontend United and wait until the 'The content is now available offline!' message appears, which means you just downloaded 672 kb of data - it is really really small, surprising no? Now switch off your network connection and reload the browser: still there! Click around and you'll be able to check the offline version at any time. If you're on a mobile device, the experience can be even sweeter: you can add this page to your homescreen, making it available as an 'app'. On iOS, you need to open the app once while still being connected to the network. We really do hope safari/iOS fixes this behavior since this is not necessary on Android. After that, turn off your network and launch the app again. Oh, and it works on a watch too if you have a browser on it. If that isn't cool, we don't know what else is! We have a little video to show you how it looks like. Watch (pun intended) and enjoy! Oh, in case we make changes to the pages, you will see a different notification telling you that the content has been updated - if your device has network of course.

Drupal integration

We've created a new project on Drupal.org, called Offline App, available for Drupal 8. The project contains the necessary code and routes for generating the appcache, iframe, pages (nodes and views) and settings to manipulate the manifest content. 3 new regions are exposed in which you can place the content for offline use. Those regions are used in offline-app-page.html.twig - but any region is available if you want to customize. Two additional view modes are created for content types and the read more link can be made available in the 'Offline teaser' mode. Formatters are available for long texts to strip internal links and certain tags (e.g. embed and iframe) and for images that will make sure that 'Link to content' is pointing to the 'Offline path'. Last, but not least, an 'Offline' Views display is available for creating lists. We're still in the process in making everything even more flexible and less error-prone when configuring the application. However, the code that is currently available, is used as is on the Fronted United website right now.

This module does not pretend to be the ultimate solution for offline content, see it as an example to quickly expose a manifest containing URL's from an existing Drupal installation for an offline version of your website. Other Drupal projects are available trying to integrate with AppCache or Service workers, however, some are unsupported or in a very premature state, apart from https://www.drupal.org/project/pwa. Note that I've been in contact with Théodore already and we'll see how we combine our efforts for coming up with one single solution instead of having multiple ones.

What about service workers ?

Not all browsers support the API yet. Even though AppCache is marked deprecated, we wanted to make sure everyone could have the same offline experience. However, we'll start adding support for service workers soon using the same concept.

We're also planning to start experimenting with delivering personal content as well, since that's also possible, yet a little trickier.

Links

Note: closed the comments for now as anti-spam modules are not doing their job nicely

Topics 

drupal, planet, offline, appcache, web application

Drupal 8 Field API series part 4: entity (form) displays and display modes

Written on December 2, 2013 - 10:14

This is part 4 of the Drupal 8 Field API series. We skipped part 3 about field types as the dust hasn't settled yet in terms of API changes. Once we get there, we'll make sure we'll talk about the massive changes that have happened in that area.

In this part, we'll talk about new API's which are, strictly speaking, not part anymore of Field API, they now belong to the Entity Field API. However, the reason the changes exist, directly comes out of a big problem in Drupal 7 in Field API: scattered storage.

Entity display

In Drupal 7, parameters controlling how to render the various "parts" of an entity view in a given view mode were scattered in many separate locations:

  • in $instance['display'] for each individual (Field API) field in the bundle
  • in the 'field_bundle_settings_[entity_type]_[bundle]' variable for the 'extra fields' in the bundle
  • in other places for contrib additions that are not one of the two above (most notably field_group module)

In Drupal 8, all those parameters ("options") for all those parts ("components") are centralised in one single EntityViewDisplay configuration object - which means they can be deployed with CMI - for each entity bundle and view mode. You can access these objects with the entity_get_display() function. All available methods can be found in EntityDisplayInterface.

<?php
entity_get_display('node', 'article', 'default')
  // Set 'component_1' to be visible, with weight 1.
  ->setComponent('component_1', array(
    'weight' => 1,
  ))
  // Set 'component_2' to be hidden.
  ->removeComponent('component_2')
  ->save();
?>

Also new in D8 is that all entity and field rendering hooks now receive this EntityDisplay object as a parameter. This means you have direct access to the configuration in your functions, instead of having to figure them out by calling functions to get the definition of field instances, extra fields or other configurations. For the full list of all hooks affected, scroll to the resources part at then end of this article for the change record link.

<?php
function hook_node_view(Node $node, EntityDisplay $display, $view_mode, $langcode) {
  if ($display->getComponent('mymodule_addition')) {
    $node->content['mymodule_addition'] = array(
      '#markup' => mymodule_addition($node),
      '#theme' => 'mymodule_my_additional_field',
    );
  }
}
?>

Note that currently only Field API fields and extra fields are handled by the EntityViewDisplay object. This is going to change when we introduce component type plugins so everyone will be able to store and access configuration. For more information, follow https://drupal.org/node/1875974.

Entity form display

Just like with the display settings, form settings are scattered all over the place:

  • in $instance['widget'] for each individual (Field API) field in the bundle
  • in the 'field_bundle_settings_[entity_type]_[bundle]' variable for the 'extra fields' in the bundle
  • in other places for contrib additions that are not one of the two above (most notably field_group module)

In Drupal 8, all those parameters ("options") for all those parts ("components") are centralised in one single EntityFormDisplay configuration object for each entity bundle and form mode.This object uses the same EntityDisplayInterface as the EntityViewDisplay object. The consequence of this API is the birth of the counterpart of view modes: form modes.

The direct use case for this was the hacky workaround in Drupal 7 for showing Field API fields on the user registration page. In Drupal 8, you can now configure which fields are available on the user registration page or the edit page. However, more interesting possibilities with this new API will certainly pop up when Drupal 8 is released, most notable inline entity forms.

UI impact

In Drupal 7, the place to manage the fields and control the order in forms is done on the 'manage fields' screen. In Drupal 8, this has has been split out in the 'manage fields' and the 'manage form display' screens. Widget settings are now also managed on this new screen, which allows you to have different widget settings depending on the form mode. The form displays screen also have a 'hidden' region, which allows you to hide certain fields on certain form modes.

Display modes

Drupal 7 has the concept of 'view modes', previously build modes in D6. In D8, we now also have form modes and both of them are configuration entities. Drupal 8 also ships with a UI to manage these display modes. You do not have to rely anymore on contributed modules like 'View modes' or 'Display Suite'.

Resources

Topics 

Drupal 8 Field API series, entity display, display mode, field ui, planet

An open source app for DrupalCon Prague

Written on September 19, 2013 - 07:56

DrupalCon Prague is approaching fast. In a few days, Drupal folks from all over the world will gather together in Prague to sprint, attend sessions, have fun etc. A few days ago, I was searching to see whether there was a mobile application available and turns out it wasn't. So me and a friend volunteered to create one based on the DrupalCamp app we've created a few weeks ago. They might be basic, but they do the job fine when you quickly want to check the schedule of the whole week.

The apps will be available for Android and iOS. The Android version is available on Play Store, the iOS version is currently in review, so fingers crossed!

Features include:

  • Built with speed in mind: native app
  • Works offline
  • Sessions and speakers
  • Create your schedule
    Note, does not sync with your account.
  • Maps integration when online

Collaborate

The code is available on GitHub, so anyone can easily send in bug reports, interface translations or create pull requests to make the applications even better. You can of course reuse and/or extend the code for future DrupalCons.

Proudly built by @swentel, @TimLeytens and @leenvs.

Topics 

android, mobile phone, iphone, drupalcon prague, planet

Pages

Subscribe to RSS - planet

You are here

Home