Planet Drupal

Email notification is a convenient way to be updated about new content or comments. There are several modules in Drupal that already deal with this issue, and I would like to propose another solution, and highlight the advantages of using it. The new system relies on Message-notify module, which depends on the Message module - a general logging system (think activity stream, but without a fancy query builder).

Message module itself is pretty simple. You can have different message types, and when you want to create a message, you actually create a new entity called Message.

What makes message so great is:

1. You can replace tokens either by hard-coding it, or by having a callback function. So if you want to have a message like “Good day @name” -- you can either replace the @name with the user’s name, and save it to the database, or you can have a callback function that will make sure we get the most up to date user name, in case it was changed.
2. Message module is using the "translatable" feature of Field API, which means we can translate the messages. Yap, you’ve heard me right, you can easily send messages in different languages!
3. Unlike other modules that only notify you via watchdog if there was an error, and the email is lost - since we are using an Entity, we can easily save those messages, and view the message as it was sent to the user, and if needed re-send them!

Lets see it in action:

• Get the -dev version of Message and Message-notify, and enable Message-notify example module
• See the message type that was added in admin/structure/messages
• Create a few nodes, and as a different user, add comments
• If your SMTP works, you will get an email, otherwise, you can view the sent messages using Views -- click on the “Message notify example” link on the navigation block
• And now for the fancy stuff, enable Locale module, and add “French” language.
  Again, as different users with different language, post comments, and see how the message is language aware!

As you can see Message notify, doesn’t try to implement a subscription system. You can do that yourself according to your custom logic, and Flag module will probably play a big part in it.

Non-developers will need to wait a little longer for a Rules integration with Message notify (there is one already for Message itself), but developers can dive in, and start building a slim an efficient notifications system using Message.

I would also like to point that we implemented for out client in Medico.com a really nice digest-email feature, that sends you all the recent activity. Remember, since tokens can be replaced by callbacks, nothing stops us from calling views_embed_view() and sending this nice email.

2011 was a big year for us Exaltation of Larks. In addition to our regular consulting and development work, we kicked off our public training program in January of 2010 and have offered public classes on everything from Drupal fundamentals to back-end development and everything in between.

In 2011, we trained organizations in Los Angeles, Silicon Valley, Irvine and San Diego and our training clients are companies including LegalZoom, Disney Interactive, Thomson Reuters, The Annenberg Foundation and Warner Brothers; universities including UCLA, UCI and UCSB; and many Los Angeles-area creative and advertising agencies.

Training Scholarship Program

Our trainings aren’t just for big organizations, however. We want to help train as many people as possible, including unemployed job seekers and people in need, and help them become the developers, themers and architects of tomorrow. To this end, we started our training scholarship program in September.

In 2011, we gave away seats at our trainings worth more than $10,000 to our scholarship students and to local area Los Angeles Drupal user groups to raffle off at their meetups. This has been a tremendously rewarding experience for us and we look forward to doing more of the same in 2012.

Upcoming Trainings

Our first training of the new year is on Drupal Scalability and Performance and it’s at SANDcamp, the San Diego Drupal Camp, on January 26, 2012! If you’re interested in making Drupal go really fast, this training is for you. We’ll provide the servers you’ll get to optimize for performance and all you need to bring is your laptop.

read more

I`m parsing the responses of the Deploying Drupal survey I started a couple of months ago (more on that later)

One of the questions in the survey is "What is devops" , apparently when you ask a zillion people (ok ok, just a large bunch of Tweeps..), you get a large amount of different answers ranging from totally wrong to spot on.

So let's go over them and see what we can learn from them ..

The most Wrong definition one can give is probably :

• A buzzword

I think we've long passed the buzzword phase, definitely since it's not new, it's a new term we put to an existing practice. A new term that gives a lot of people that were already doing devops , a common word to dicuss about it. Also lots of people still seem to think that devops is a specific role, a job description , that it points to a specific group of people doing a certain job, it's not . Yes you'll see a lot of organisations looing for devops people, and giving them a devops job title. But it's kinda hard to be the only one doing devops in an organisation.

I described one of my current roles as Devops Kickstarter, it pretty much describes what I`m doing and it does contain devops :)

But devops also isn't

• The connection between operations and development.
• people that keep it running
• crazy little fellows who find beauty in black/white letters( aka code) rather than a view like that of Taj in a full moon light.
• the combination of developer and operations into one overall functionality
• The perfect mixture between a developer and a system engineer. Someone who can optimize and simplify certain flows that are required by developers and system engineers, but sometimes are just outside of the scope for both of them.
• Proxy between developer and management
• The people in charge of the build/release cycle and planning.
• A creature, made from 8-bit cells, with the knowledge of a seasoned developer, the skillset of a trained systems engineer and the perseverence of a true hacker.
• The people filling the gap between the developer world and the sysadmin world. They understand dev. issues and system issues as well. They use tools from both world to solve them.

Or

• Developers looking at the operations of the company and how we can save the company time and money

And it's definitely not

• Someone who mixes both a sysop and dev duties
• developers who know how to deploy and manage sites, including content and configuration.
• I believe there's a thin line line between Ops and Devs where we need to do parts of each others jobs (or at least try) to reach our common goal..
• A developer that creates and maintains environments tools to help other developers be more successful in building and releasing new products
• Developers who also do IT operations, or visa versa.
• Software developers that support development teams and assist with infrastructure systems

So no, developers that take on systems roles next to their own role and want to go for NoOps isn't feasable at all ..you really want collaboration, you want people with different skillsets that (try to) understand eachoter and (try to) work together towards a common goal.

Devops is also not just infrastructure as code

• Writing software to manage operations
• system administrators with a development culture.
• Bring code management to operations, automating system admin tasks.
• The melding of the art of Systems Administration and the skill of development with a focus on automation. A side effect of devops is the tearing down of the virtual wall that has existed between SA's and developers.
• Infrastructure as code.
• Applying some of the development worlds techniques (eg source control, builds, testing etc) to the operations world.
• Code for infrastructure

Sure infastructure as code is a big part of the Automation part listed in CAMS, but just because you are doing puppet/chef doesn't mean you are doing devops.
Devops is also not just continous delivery

• A way to let operations deploy sites in regular intervals to enable developers to interact on the systems earlier and make deployments easier.
• Devops is the process of how you go from development to release.

Obviously lots of people doing devops also often try to achieve Continuous delivery, but just like Infrastructure as Code it devops is not limited to that :)

But I guess the truth is somewhere in the definitions below ...

• That sweet spot between "operating system" or platform stack and the application layer. It is wanting sys admins who are willing to go beyond the normal package installers, and developers who know how to make their platform hum with their application.
• Breaking the wall between dev and ops in the same way agile breaks the wall between business and dev e.g. coming to terms with changing requirements, iterative cycles
• Not being an arsehole!
• Sysadmin best-practise, using configuration as code, and facilitating communication between sysadmins and developers, with each understanding and participating in the activities of the other.
• Devops is both the process of developers and system operators working closer together, as well as people who know (or who have worked in) both development and system operations.
• Culture collaboration, tool-chains
• Removing barriers to communication and efficiency through shared vocabulary, ideals, and business objectives to to deliver value.
• A set of principles and good practices to improve the interactions between Operations and Development.
• Collaboration between developers and sysadmins to work towards more reliable platforms
• Building a bridge between development and operations
• The systematic process of building, deploying, managing, and using an application or group of applications such as a drupal site.
• Devops is collaboration and Integration between Software Development and System Administration.
• Devops is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals.[1] It has developed in response to the emerging understanding of the interdependence and importance of both the development and operations disciplines in meeting an organization's goal of rapidly producing software products and services.
• bringing together technology (development) & content (management) closer together
• Making developers and admins understand each other.
• Communication between developers and systems folk.
• a cultural movement to improve agility between dev and ops
• The cultural extension of agile to bring operations into development teams.
• Tight collaboration of developers, operations team (sys admins) and QA-team.

But I can only conclude that there is a huge amount of evangelisation that still needs to be done, Lots of people still don't understand what devops is , or have a totally different view on it.

A number of technology conferences are and have taken up devops as a part of their conference program, inviting experienced people from outside of their focus field to talk about how they improve the quality of life !

There is still a large number of devops related problems to solve, so that's what I`ll be doing in 2012

A quick follow-up to the last blog post to mention that the first stable release of Nodewords in two years, v6.x-1.13, has finally been released. A huge thank you is due to the few people who helped to test it over the holidays, without whom this release would not have been quite as stable.

Tags:

• Drupal
• Drupal Planet

We recently needed to migrate all our sites on one physical server to another server, there were more than 200 sites, and they were all hosted with Aegir. The old server was to be decommissioned, so we had to move all of Aegir's data about the site to the new server import into a new Aegir master on the new server. We also needed to do this with as small amount of downtime as possible.

In the end we migrated all the sites with about 30 seconds of downtime each, here's how:

The setup before

For clarity, here's a simplified diagram of the infrastructure before the migration:

You can see we have two servers behind a firewall, the firewall basically is there to translate the public IP address into an internal IP address of the actual web server.

We set up the two webservers to be as identical as possible, in terms of Aegir installed and the names of the platforms hosting the sites on them etc. but this process is actually quite flexible, so for example, we actually had a dedicated DB server on our old infrastructure, but not in the new.

We installed the code I'm about to describe on both machines, into ~aegir/.drush/ and additionally we ensured that the aegir user on the new machine could ssh into the old machine as the aegir user with its ssh key. We also created a Drush alias for the 'old_hostmaster' by creating the file: ~aegir/.drush/aliases.drushrc.php and adding the entry:

$aliases['old_hostmaster'] = array(
  'remote-host' => '192.168.1.2',
  'remote-user' => 'aegir',
  'uri' => 'old-hostmaster.computerminds.co.uk',
  'root' => '/var/aegir/hostmaster-6.x-1.5',
);

(You will need to set the details to match your environment.)

Reducing downtime

To reduce the amount of downtime during the server migration, we employed the apache module: mod_proxy. This allows the apache server on the new webserver to function as a reverse proxy server. Traffic arriving at the new server gets forwarded to the old one and visitors don't notice any difference.

Setting up 200 or more virtual host files with all the correct details is a pain though, and so naturally we wrote a Drush command to do it for us.

First we need to get a list of sites on an Aegir platform, which we can do like this:

/**
 * Lists all sites from a particular platform.
 *
 * Prints a serialized array of URLs.
 */
function drush_computerminds_migrate_platform_list_sites($platform) {
  $sites = array();

  $platform_node = hosting_context_load($platform);

  // Load the platform:
  $all_sites = hosting_get_sites_by_status($platform_node->nid, HOSTING_SITE_ENABLED);

  foreach ($all_sites as $site) {
    $sites[] = $site->title;
  }
  
  // This is a bit of a fragile way to return the data, but I couldn't seem to
  // get Drush to pass the structured data back properly, so we'll do this,
  // which works.
  drush_print_r(serialize($sites));
}

Because this is a Drush command itself, we can call this on a 'remote' server. So we can run a command from the new server, and go get all the sites we need to migrate from the old server.

/**
 * Sets up the current hostmaster ready for migration.
 */
function drush_computerminds_migrate_pre_migrate_setup() {
  $platform = COMPUTERMINDS_PLATFORM_NAME;
  $old_hostmaster = '@' . COMPUTERMINDS_OLD_HOSTMASTER_NAME;
  $platform_context = d('@platform_' . $platform);
  $web_server = d($platform_context->web_server);

  $old_sites = computerminds_migrate_get_all_sites($old_hostmaster, $platform);

  foreach ($old_sites as $site) {
    // Need to ensure that we have a mod_proxy vhost for this site.
    drush_log('Creating a mod proxy vhost for: ' . $site);
    $vhost = new provisionConfig_computerminds_proxy($web_server, array('uri' => $site));
    $vhost->write();
  }

  // Now restart the web server.
  $web_server->service('http')->restart();
}

/**
 * Get all sites of a given platform on a given server.
 */
function computerminds_migrate_get_all_sites($target, $platform) {
  $platform = 'platform_' . $platform;
  // Get a list of all the sites on the remote hostmaster.
  $result = drush_backend_invoke_args('@' . ltrim($target, '@') . ' ' . 'platform-list-sites', array($platform), array('root' => NULL, 'uri' => NULL), 'GET', FALSE);
  $sites = unserialize($result['output']);
  if (is_array($sites)) {
    return $sites;
  }
  return array();
}

This is a fairly straightforward command in which we go an get a list of sites on a particular platform on the old server, and then create a provisionConfig_computerminds_proxy for site and write it. The provisionConfig_computerminds_proxy class looks like this:

/**
 * Base class for proxied virtual host configuration files.
 */
class provisionConfig_computerminds_proxy extends provisionConfig {
  public $template = 'computerminds_proxy_vhost.tpl.php';
  public $description = 'mod proxy virtual host configuration file';


  function filename() {
    return $this->http_vhostd_path . '/' . $this->data['uri'];
  }

  function process() {
    parent::process();

    $this->data['http_port'] = $this->http_port;
    $this->data['http_proxy_forward'] = COMPUTERMINDS_OLD_SERVER_IP;

    if ($this->aliases && !is_array($this->aliases)) {
      $this->aliases = explode(",", $this->aliases);
    }
    else {
      $this->aliases = array();
    }

    $this->aliases = array_filter($this->aliases, 'trim');

    $uri = $this->data['uri'];
    if (strpos($uri, 'www.') === 0) {
      $this->aliases[] = substr($uri, 4);
    }
    else {
      $this->aliases[] = 'www.' . $uri;
    }
  }
}

This class just sets up some basic Aegir config stuff, and requires a simple template, computerminds_proxy_vhost.tpl.php:

<VirtualHost *:<?php print $http_port; ?>>
  ServerName <?php print $this->data['uri']; ?>

  <?php
  if (sizeof($this->aliases)) {
    print "\n ServerAlias " . implode("\n ServerAlias ", $this->aliases) . "\n";
  }

  print " RewriteEngine on\n";
  foreach ($this->aliases as $alias) {
    print " RewriteCond %{HTTP_HOST} ^{$alias}$ [NC]\n";
    print " RewriteRule ^/*(.*)$ http://{$this->data['uri']}/$1 [L,R=301]\n";
  }

  ?>

  ProxyRequests Off
  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>

  ProxyPass / http://<?php print $http_proxy_forward; ?>/
  ProxyPassReverse / http://<?php print $http_proxy_forward; ?>/
  ProxyPreserveHost On

</VirtualHost>

We ran this Drush command before starting the migration, and actually because our new server had a public IP address of its own, we could test beforehand that access one of the sites domains at that IP, we actually accessed the site on the old server.

Switching IPs

Once we had our servers set up we were then good to start the migration process, the first stage was to re-assign the public IP address of the sites to the new server, instead of being attached to the old, this was the traffic flow:

Note, that if you don't have the ability to re-assign public addresses between servers, then you could switch the DNS entries over to the new IP, and just wait until all the traffic is hitting the new server and not the old one.

Doing the actual migration

The actual migration process is pretty simple, we take advantage of Aegir's built-in Drush commands to do all of the heavy lifting.

Here is the complete Drush command to migrate one site from the old server to the new one.

/**
 * Drush command to migrate a single site from the old server to the new one.
 */
function drush_computerminds_migrate_migrate_computerminds_one($site) {
  // Do a backup on the old server.
  drush_log(dt('Backing up old site: @uri', array('@uri' => $site)), 'ok');
  $suggested = d()->platform->server->backup_path . '/' . $site . '-migrate-' . date("Ymd.His", mktime()) . '.tar.gz';
  drush_backend_invoke_args('@' . ltrim($site, '@') . ' ' . 'provision-backup', array($suggested), array('uri' => $site, 'root' => d(COMPUTERMINDS_PLATFORM_NAME)->root), 'GET', TRUE, NULL, COMPUTERMINDS_OLD_SERVER_IP, 'aegir');

  // Rsync to this machine.
  drush_log('Copying backup from remote server...', 'ok');
  if (drush_core_call_rsync(escapeshellarg('aegir@' . COMPUTERMINDS_OLD_SERVER_IP . ':' . $suggested), escapeshellarg($suggested), array(), TRUE, FALSE)) {
  }
  else {
    return drush_set_error('RSYNC_FAILED', 'Failed to copy the backup from the remote server.');
  }

  // Copy the Aegir context file over.
  $alias_file = '/var/aegir/.drush/' . $site . '.alias.drushrc.php';
  if (drush_core_call_rsync(escapeshellarg('aegir@' . COMPUTERMINDS_OLD_SERVER_IP . ':' . $alias_file), escapeshellarg($alias_file), array(), TRUE, FALSE)) {
    // Now set the new DB server
    $args = array(
      'uri' => "$site",
      "@$site",
      'db_server' => '@' . COMPUTERMINDS_NEW_DB_SERVER,
      'root' => d('platform_ ' . COMPUTERMINDS_PLATFORM_NAME)->root,
      'platform' => '@platform_' . d(COMPUTERMINDS_PLATFORM_NAME)->name,
    );
    drush_backend_invoke('provision-save', $args);
  }
  else {
    return drush_set_error('RSYNC_FAILED', 'Failed to copy the alias from the remote server.');
  }

  drush_log('Copied all files from remote server.', 'ok');

  // Deploy the site.
  provision_backend_invoke($site, 'provision-deploy', array($suggested), array('old_uri' => $site));
  drush_log('Deployed the files and database locally.', 'ok');

  // Import into the frontend, if there are no errors.
  if (!drush_get_error()) {
    drush_log('Importing the site into the frontend...', 'ok');
    provision_backend_invoke('@hostmaster', 'hosting-import', array("@" . $site));
    provision_backend_invoke('@hostmaster', 'hosting-task', array("@" . $site, 'verify'));
    provision_backend_invoke('@hostmaster', 'hosting-task', array("@" . $site, 'enable'));
    drush_bootstrap(DRUSH_BOOTSTRAP_DRUPAL_LOGIN);

    // Hosting will create a dummy install task, but it'll fail. So we remove it here.
    $ref = hosting_context_load("@" . $site);
    if ($ref->nid) {
      if ($task = hosting_get_most_recent_task($ref->nid, 'install')) {
        drush_log(dt('Removed the dummy install task: @nid.', array('@nid' => $task->nid)), 'ok');
        _computerminds_migrate_node_delete($task->nid);
      }
    }

    drush_log(dt('The site: @uri has been imported.', array('@uri' => $site)), 'ok');
  }
}

Note that we copy the context from the old server, so that settings stored about the site are also copied over to the new server. We reset a few of those, such as the database server.

We wanted our site's Aegir generated vhost to be used in place of the proxied one we created earlier, so we implemented a couple of Drush hooks to hook into the above command and remove the proxy vhost when it is run, and put it back if the command to migrate the site fails:

/**
 * Implements drush_hook_pre_migrate_computerminds_one().
 *
 * We use the pre command hook to remove our temporary vhost.
 */
function drush_computerminds_migrate_pre_migrate_computerminds_one($site) {
  // Remove the proxy vhost
  $platform = COMPUTERMINDS_PLATFORM_NAME;
  $platform_context = d('@platform_' . $platform);
  $web_server = d($platform_context->web_server);
  drush_log('REMOVING mod proxy vhost for: ' . $site);
  $vhost = new provisionConfig_computerminds_proxy($web_server, array('uri' => $site));
  $vhost->unlink();
  $web_server->service('http')->restart();
}

/**
 * Implements drush_hook_pre_migrate_computerminds_one_rollback().
 *
 * We use the pre rollback command hook to replace our temporary vhost if
 * something when wrong with this migrate.
 */
function drush_computerminds_migrate_pre_migrate_computerminds_one_rollback($site) {
  // Add the proxy vhost.
  $platform = COMPUTERMINDS_PLATFORM_NAME;
  $platform_context = d('@platform_' . $platform);
  $web_server = d($platform_context->web_server);
  drush_log('Creating a mod proxy vhost for: ' . $site);
  $vhost = new provisionConfig_computerminds_proxy($web_server, array('uri' => $site));
  $vhost->write();
  $web_server->service('http')->restart();
}

Finally, to be able to migrate all the sites we have a simple Drush command that gets a list of sites that could be migrated and compares that to the sites on the current server, and offers to migrate them:

/**
 * Drush command to migrate a lot of sites from an old Hostmaster to this one.
 */
function drush_computerminds_migrate_migrate_computerminds_all() {
  // Get the remote sites.
  $remote_sites = computerminds_migrate_get_all_sites('@' . COMPUTERMINDS_OLD_HOSTMASTER_NAME, COMPUTERMINDS_PLATFORM_NAME);

  // Get the local sites.
  $local_sites = computerminds_migrate_get_all_sites('@hostmaster', COMPUTERMINDS_PLATFORM_NAME);

  // Migrate the diff.
  $sites_to_migrate = array_diff($remote_sites, $local_sites);
  
  $limit = drush_get_option('limit', 0);
  // Truncate the list of sites to migrate if there is a limit.
  if (!empty($limit)) {
    $sites_to_migrate = array_slice($sites_to_migrate, 0, $limit);
  }

  drush_log(dt('The following sites will be migrated:'), 'ok');

  foreach ($sites_to_migrate as $site) {
    drush_log('  ' . $site, 'ok');
  }

  if (!drush_confirm(dt('Do you want to proceed.'))) {
    return;
  }

  drush_log(dt('Migrating...'), 'ok');

  $success = array();
  $failed = array();
  
  $count = 0;

  foreach ($sites_to_migrate as $site) {
    $result = provision_backend_invoke('@hostmaster', 'migrate-computerminds-one', array($site));
    $count++;
    drush_log(dt('@count of @total sites migrated...', array('@count' => $count, '@total' => count($sites_to_migrate))), 'ok');
    if (!empty($result['error_status'])) {
      $failed[] = $site;
    }
    else {
      $success[] = $site;
    }
  }

  if (!empty($success)) {
    drush_log(dt('The following sites migrated successfully'), 'ok');
    foreach ($success as $site) {
      drush_log('  ' . $site, 'ok');
    }
  }

  if (!empty($failed)) {
    drush_log(dt('The following sites migrated unsuccessfully'), 'error');
    foreach ($failed as $site) {
      drush_log('  ' . $site, 'error');
    }
  }
}

We had to add a --limit option because Drush creates so many log messages that the parent Drush process asking all the other processes to do the work ran out of memory.

The actual command we ran to do the migration was:

drush @hostmaster migrate-computerminds-all --limit=50

And we just kept running it until all the sites had been migrated. Each site will in turn be taken off-line, by removing the proxy vhost, and then backed up, migrated, and imported onto the new server. This took about 30 seconds per site for us. After a few hours or so our setup looked like this:

We were then free to remove the old server after confirming that no traffic was being routed to it. It should be noted that apart from installing this code onto the old server, we've not actually changed anything on it, so if something goes wrong we could just switch the IP back in the firewall and go back to serving sites from the old server until we fixed the problem.

The code

I've put the code described in this article into a github repository:

https://github.com/computerminds/aegir_sites_migrate

I've cleaned it up and removed names for the purpose of this article, which may mean that I've broken it somewhere along the way. Feel free to fork and send a pull request with any fixes.

You will want to edit the defines at the top of the code to set things up.

We want to use ctools' modal frames and field collection forms to create a better user experience.

As we know, ctools comes with a lot of useful apis and tools to use in our own modules. One of them is the modal frames.

Read moreSubmitted by Dominique De Cooman on Fri, 2011-12-30 08:50ctools modal framesDrupal 7 Using ctools' modal frames and field collection forms to create a better user experience

read more

The Drupal Association is about to hold its first open elections of "at-large" (community-elected) board members--and your help is needed. Since these are community positions, we felt that it was important to get community buy-in to the process used. We need YOUR input on these suggestions to help come up with a fair and equitable voting process.

We're looking both for candidates and for input into the election process. Read on to find out how you can get involved.

Background

This discussion is meant to create a process for electing the two at-large members of the US-based 501c3 Drupal Association Board.

read more

The screencast goes through the installation of high performance Barracuda Aegir stack on Ubuntu. It's going to install and configure Nginx, PHP-FPM 5.2.17, MariaDB or Percona and other necessary components. All you need is a freshly installed server and the script will take care of the rest. You'll also find out about a couple of issues you may run into.

Watch the Screencast

The Drupal Association is holding its first round of elections for board members under its new governance structure. See the the elections announcement for background and details. To invite community input on how best to run the elections, we've outlined below some possible suggestions, along with their pros and cons. Please tell us what you think by posting comments, or add your ideas!

What aims should a voting system meet?

To clarify our best options in an election system, it may help to capture goals. What are we trying to achieve?

• Meet the requirements as set out in the DA bylaws and Election Committee charter.
• Provide a basis to evaluate the success of this initial round of elections and provide feedback and recommendations.

What should be required for nomination?

One facet of the election process that needs to be decided is how nominations happen. Should prospective board members nominate themselves, or should they be nominated by the community? Or some combination thereof?

Note that existing Drupal Association board members or members of the DA's advisory board are not eligible for election to at-large seats.

Here are some options, as well as advantages and disadvantages:

Self nomination only

Details: Nominations come from individuals willing to run themselves

Advantages

• This eliminates the possibility of adding someone to the ballot who has popular support of the community, but who is unwilling or unable to serve on the board of the Drupal Association.

Disadvantages

• May lead to candidates that do not have support of the larger community

Nomination by others

Advantages

• Would help identify people the community would like to see on the board.

Disadvantages

• Could end up nominating people who do not want to serve.
• Requires a vetting process of nominees so votes do not get cast for invalid candidates.
• May lead to many candidates.

Nomination by defined Drupal community entities

Entities could include:

• Drupal User Groups
• Teams such as security, documentation, infrastructure

Advantages

• Would help identify people the community would like to see on the board.

Disadvantages

• Drupal entities may not be set up to make such decisions, especially to do so quickly.
• Could end up nominating people who do not want to serve.
• Requires a vetting process of nominees so votes do not get cast for invalid candidates.

Nomination by self AND/OR others

Candidate can nominate themselves, but must be seconded by someone else. Candidate, if nominated by others, must accept nomination.

Advantages

• eliminates the possibility of adding someone to the ballot who has popular support of the community, but who is unwilling or unable to serve on the board of the Drupal Association.
• help identify people the community would like to see on the board.

Disadvantages

• None. Combines the advantages of two of the other options, and mitigates against the disadvantages of self nomination only or nomination by others.

Qualification to vote

Obviously, one of the most important aspects to elections is who has the right to vote on these members. There are many facets that could be used to determine whether someone is a member of the community or not; some easier to check than others.

Here are some possible ideas we could use to identify voters, as well as their pros and cons. Other ideas welcome!

Is an individual member of the Drupal Association by a certain date.

Advantages

• Guarantees that the person is real
• Cost for becoming an individual member is relatively inexpensive (inexpensive for developed countries, but expensive for members in developing countries such as China or India)
• is not overly exclusive

Disadvantages

• DA membership has yet to have any defined rights and, to this point, has only been cast as a way to support the work of the Association. There may be significant pushback from the community along the lines of pay-to-play.

Is a member of drupal.org

Advantages

• Broadly inclusive

Disadvantages

• it's very hard to verify that these are "real people" and not fake accounts.
• Possible to tarnish the election results by bulk-registering accounts. Though one option could perhaps be to put a minimum time on the accounts, e.g. must be a d.o user for >= 1 year.

Has attended a DrupalCon in the last year

Advantages

• shows active involvement and interest in Drupal

Disadvantages

• would exclude people whose input we value who were busy in the last year and/or cannot afford the travel and entry to the event.

Has attended a Drupal meetup

Advantages

• shows active involvement and interest in Drupal

Disadvantages

• very hard to track unless we go by self-reporting in which case it's hard to tell they are not fake accounts

Has made commits to Git

Advantages

• Rewards people who help make our software awesome with a say in the board of the Drupal Association

Disadvantages

• Leaves out significant portion of non-developer community members from the voting process.
• Leaves out important contributors to other aspects of the project, e.g. Documentation, User Experience.
• Not all commits are equal

Has contributed to drupal.org

Criteria could include:

• Has contributed documentation (link to page contributed prior to a given date).
• Is a manager of a group on groups.drupal.org (link to group).
• Maintains a project (link to project page).

Advantages

• Recognizes various ways to contribute to Drupal.

Disadvantages

• Relatively difficult to verify--would require individual confirmation of each voter.

How should the community interview candidates?

Once we have the list of nominees, the community needs to be given a chance to ask them questions, find out more about their stances on various issues and particular initiatives they wish to spear-head as part of the board.

Here are some ideas. Of course, these are not mutually exclusive and - time and resources allowing - we could use any or all.

Comments enabled on candidate submission forms

(e.g. "Nomination" node form with comments turned on)

Advantages

• Allows drilling in to one candidate to learn more about them.

Disadvantages

• questions (accusations, recriminations, etc.) are directed toward a single candidates.

questions should be asked of all candidates and be open to response from all candidates

Advantages

• having an open forum will, if nothing else, give a measure of a candidate's willingness and ability to express theirself

Disadvantages

• Might get uneven coverge of candidates, depending on availability.

All candidates meeting

Have a meeting on IRC/phone that allows for "real time" questioning of candidates by community.

Advantages

• More immediate feedback on questions
• Better chance to "get to know" prospective candidates.

Disadvantages

• Leaves out people who cannot be there a given time.
• Technology may leave some members of the community out entirely.

Where should the elections take place?

drupal.org

Advantages

• Easily accessible

Disadvantages

• Requires vetting a new module for deployment on Drupal.org, will delay election process
• About 100 people, likely some of whom will be on the candidates list, have administrative access

association.drupal.org

Advantages

• Already has webform module deployed
• Fits well, since the elections are for the Drupal Association
• Easily accessible

Disadvantages

• Off the "main" site
• About 25 people have administrative access (though likely none who are eligible for candidacy)

A subsite for atlarge.association.drupal.org?

Advantages

• Single purpose site with the functionality and security required to run the election.
• Can install relevant modules without worrying about impact on the functioning of D.O or A.D.O
• Becomes an historical record of board elections and outcomes.
• Potentially could be used for other democratic and transparency activities such as plebiscites, referendums, committee elections.

Disadvantages

• We have to build and maintain a whole site.

Who should carry out the elections, e.g., count votes?

• Only a small handful of extremely trusted people (d.o infrastructure team? DA election committee?) should be able to view who voted for whom. This preserves the ability for people to vote their conscience, even if that means voting against their boss or a co-worker/friend.

How should voting work?

Then, finally, on the voting process itself. Here's some ideas, but we welcome others as well.

First past the post (FPTP)

Every qualified voter gets up to two votes of equal weight. They are not permitted to cast more than one vote for a single candidate. The two candidates with the most votes (simple majority) are elected.
For more information on this voting system see http://en.wikipedia.org/wiki/First-past-the-post_voting

Advantages

• No need to sift through votes and perform calculations on first vs. second choices or similar voting mechanisms, a simple, anonymized count would be sufficient.

Disadvantages

• A candidate with the MOST votes doesn't necessarily represent the MAJORITY of voters.
• People tend tol vote for candidates most likely to win to prevent wasting their vote.
• FPTP is a widely criticised voting system.

Instant Run Off Vote / Preferential

Voters rank all candidates in order of preference. The candidate with the least number of votes is eliminated, and the 2nd preference of those votes is transferred to the other candidates, and so on, until one candidate receives an absolute majority - or 50% +1 of all votes.
For more information on this voting system see http://en.wikipedia.org/wiki/Instant-runoff_voting

Advantages

• Ensures the outcome is more representative of the will of more voters.

Disadvantages

• It is more complex to calculate the outcome (Note: Decisions module has implemented an instant run off algorithm)

Questions for candidates

What information should we ask of candidates? This might form the basis e.g. of an information sheet that all candidates are asked to fill in, with their responses being publicly accessible as a basis for evaluating them as candidates.

• Name
• Where located (e.g., city and country)
• drupal.org user name, if applicable

Timeline

In order to get new board members ramped up in time for DrupalCon Denver, the first in-person board meeting of 2012, we need to adhere to the following schedule:

• Have an open nomination period of 1 week. Potential board candidates submit a form with a description of their qualifications and motivations for applying.
• Voting period of 1 week. (see below for who gets to vote).

• 2 candidates are presented to the board for ratification.

• December 30- January 6 (2 weeks): Voting proposal open for public comment (this post).

• January 9 - January 13 (1 week): Drupal Association's Election Committee takes community feedback and prepares final voting proposal for Board of Directors, sets up whatever tools are required for election process.
• January 18: Voting proposal submitted to BoD for ratification (cross-posted to g.d.o). Assuming this is accepted....
• January 19 - January 27 (10 days): Nomination process.
• January 29 - February 7 (10 days): Election process.
• February 8: Elected at-large members submitted to board for ratification.

That means we need feedback as soon as possible!

TL;DR (aka, The Short Version)

We welcome community thoughts on the following questions as they pertain to filling two community-elected board positions for the Drupal Association:

• How should nominations work? Should interested parties self-nominate for an at-large position, or should we take suggestions from the larger community? Other?
• Who should be able to vote during elections? Anyone with a Drupal.org account? Anyone who's a paid Individual Member of the Drupal Association? Or some other criteria?
• What should the community "vetting" process of prospective candidates look like? Comments on individuals' applications? A forum of some kind for the community to raise questions/issues and offer all candidates an opportunity to respond? Some kind of "real-time" meeting?
• Where and how should the actual elections themselves take place? Webform module, locked down to only a select set of members to view results? A public voting process on Twitter or IRC? What are your ideas?

Thanks very much for your participation in a very important step for the Drupal Association's community accountability!

Drupal Association

It can be difficult to remember all of the usernames and passwords that you use to log in to various websites across the internet, so why force users to create a new username for your web site? It's easier on everyone to simply combine the username and email address fields. It also cleans up your registration form a bit.

In Drupal, there are two modules that can help you to accomplish this:

• Email Registration
• Logintoboggan

These two modules are mutually exclusive— they are not compatible with each other, so you'll have to pick one. I prefer to use Email Registration, and I'll explain why.

Email Registration

Email Registration simplifies the user registration form by removing the 'username' field. By default, the module will automatically generate a username for a new user based upon the first part (before the @) of his/her email address. For example, if I registered for a new account with email address madmatter23@grasmash.com, I would end up with username madmatter23.

That's great, but it's not exactly what I'd like. I'd like my username to be my full email address. Luckily, Email Registration provides a hook_email_registration_name() to let you customize exactly how the username will be generated. I used the hook in this way:

/* 
 * Implements hook_email_registration_name().
 */
function grasmash_email_registration_name($edit, $account) {
  return $account->mail;
}

Adding a similar snippet to your own custom module will give you complete control. That's all there is to it!

LoginToboggan

This module still forces users to choose their own username during registration. However, after they're registered, it will allow them to login using either their username or their email address. Hence my preference for Email Registration.

LoginToboggan also has a number of other nice features, such as redirecting users after registration or login, and providing a login form on the 403 Access Denied page. If you really need one of these features, there are a number of alternative modules that provide the same functionality:

• Redirect 403 to User Login — 'nuff said.
• Rules — this can be used to do a variety on the login or registration events, including a redirect.

Final Thought

Using this method does have at least one major drawback: you no longer have the option to display a user's username while preserving the privacy of their email address. However, this problem can easily be circumvented by simply using an optional 'alias' field, or by utilizing a programmatically applied handle for users based on other field values.

6.x, , 7.x, drupal, login, registration

The Wysiwyg module for Drupal already provides quite a few tools for the word processor based editors to be able to create Word-esque types of enriched content in their site. Even with those tools though, their is still desire for the "point-and-click" webmaster to be able create even more enhanced elements without having to, well, know code. The combination of the Wysiwyg module and Drupal provides methods of adding plug-ins to the Wysiwyg editors.

For the purposes of this post, the methods to be able to create tabbed and accordion drop-down style content via the CKEditor editor will be covered.

Many projects these days keep their work in github and it is a request often to make some dev site automated automatically. When I tried to enter a passworded URL into github, it didn't work. So I came up with what's below.

read more

Within the last couple of months one of our client’s websites was hacked. This is a rarity, and we responded swiftly to restore a previous version of the site; however, we still needed to get rid of the actual problem. Through the experience there were a number of lessons learned that anyone, whether client or developer could benefit from. 

read more

I've been facilitating the Drupal core office hours for about four months now. You know those sorta-similar tweets you see retweeted from @drupalcore most Wednesdays? Yeah, that chirpy little bird is me.  Now that D8 development is picking up, D7 is approaching its first birthday, and DrupalCon Denver is right around the corner, I'd like to share some updates. What follows is some background on office hours, a status report of sorts, and my pitch for a "core office day" sprint at DrupalCon Denver. (Seeking co-leads!)

What are Drupal core office hours?

The Drupal core office hours have a simple goal: Help people help with core.  If you've missed out so far, here's the skinny:

• Office hours are held twice a week in the #drupal IRC channel:
  • Tuesdays 0:400 - 0:600 UTC (Monday evening in North America; Tuesday in Europe and Asia)
  • Wednesdays 16:00 - 18:00 UTC (morning in North America; afternoon in Europe; late night in Asia)
• People of all experience levels are welcome to participate.
• Experienced core contributors help each participant find and work on tasks.
• Typical office hours tasks include:
  • Co-authoring change notifications and issue summaries.
  • Triaging issues (including finding steps to reproduce, clarifying the issue's status, locating duplicates, closing those that are no longer relevant, etc.).
  • Testing patches that need manual testing.
  • Adding automated tests for existing patches.
  • Rerolling patches.
  • Exchanging patch reviews.

Why do we do this?

• Participating in the Drupal core issue queue for the first time can be confusing or intimidating.
• Providing a real-time way to contribute helps connect new contributors with the Drupal community as well as with the issues.
• Talking about core development in #drupal helps open the community's "silos".
• There are over 9000 open issues filed against Drupal core.
• Many of these open issues simply need triaging.
• Many others will move forward with summaries, testing, review, etc.
• Core has a chronic shortage of patch reviewers.

How has it been going?

I'm usually asleep during the Tuesday timeslot, but I've been keeping track of office hours participation on Wednesdays.  A few interesting statistics:

• 47 different people have participated in the Wednesday timeslot alone.
• These people have worked on 71 separate core issues.
• Experience levels have ranged from novices participating in a core issue for the first time to active core developers and even webchick. ;)
• There have been participants on 16 of 20 Wednesdays (80%).
• The highest number of participants on a single day so far was seven on Sept. 28.  (That was a busy day for me.)
• Many new contributors go on to work on the core queue on their own time after they participate office hours.

What can we improve?

• A stated goal is for office hours to be self-sustaining.  This hasn't really happened yet; the one week I was unavailable on a Wednesday, there weren't any office hours.
• Attendance has tapered off over the past two months; there have been no attendees for four of the past seven weeks.  I see two possible causes:
  1. The holidays.
  2. The time change.  (The UTC timeslots are an hour earlier for the Northern hemisphere now that daylight savings time has ended).
• The Tuesday slot has been somewhat quiet and sporadic, and might benefit from more activity.  I've spoken to a few people who have want to participate, but can't make the Wednesday time.  (We could particularly use another facilitator who can commit to being available regularly during the Tuesday timeslot.)
• Having a prepared list of possible tasks helps things go more smoothly and makes it easier to "recruit" out of #drupal.
• It would be beneficial to have a couple participants who work on core regulary each week, if only to commit to an hour or two of issue queue work and talk about what they're doing in the channel.
• And, of course, more participants.  I've noticed that at a certain point the office hour activity reaches a "critical mass" where other people in the channel ask what's going on and end up participating.

How can I help?

• Come to office hours!
• Tell your friends and coworkers about office hours.  Mention them on your blog, twitter, or IRC when you see someone asking how to contribute.
• Show my Core and you slideshow at your local Drupal user group.

Denver core office day!

I'd like to propose a sprint for Drupalcon Denver that takes the core office hours idea and expands it into a whole day.  I'd imagine:

• A bit of a "core queue orientation" for those just getting started in core development.
• Talking about how to do a good patch review, write a good summary or change notice, etc.
• A list of targeted issues in various categories.
• Prioritizing backportable issues.
• A place where people could get help with setting up a LAMP stack, git, etc.  (Actually, it would be great to have this station on the code sprint day regardless.)
• Novices and experienced contributors who don't necessarily want to stay all day dropping in for an hour or two to chip in.
• Maybe a fun counter of the number of issues triaged, summarized, reviewed, etc., and before-and-after snapshots of the core queue. :)

I'd love to hear your ideas or suggestions; just leave a comment.  I also will need some sprint co-leads if I propose this.  (It's a required field on the sprint proposal form.)  cweagans has already said he'd be interested.  Anyone else?  A good co-lead would have some experience doing patch reviews, know the core queue's "rules," and be willing to help novices get oriented and find tasks.

 

The next few days are like a timer counting down. Sometimes the conversations feel a bit like a NASA control room. We review and re-review our plans because this deployment isn't going to be a small one. It is likely in the top 3 in scope since I began over two years ago.

Day 12 - Testing

We started with Scrum, Scrum, Scrum, and Scrum of Scrums.

Testing continued on Day 12 - and all the while we continued to work through Product Manager user stories. Supporting artifacts were shared as well. This segued into the Project Managers reviewing points and starting to assign teams for the next timebox. For this cycle, we found that our story points were roughly double to the number of the points available for the next timebox. This requires a feedback loop with the Executive committee to prioritize completed set of user stories.

The Executive Technical Leadership Committee met where we discussed a variety of issues, successes, and follow ups from the prior week. This committee fulfills the role of a CTO.

Finally, I also reviewed a patent application.

Day 13 - Testing, Stories, and Horse Trading

We continued to work through user stories and test. We had a 90 minute meeting with the Executive Committee to discuss the priorities where some items were shifted lower on the priority list and, for intents, got knocked out of the upcoming timebox and into the next.

Testing is coming to an end at this point. Day 14 will be release day and so only the most critical items are prioritised to be addressed before we release. This means several status meetings throughout the day. In this timebox, we were still engaged with status meetings until later in the evening. This was partially due to a timebox that was 25% smaller than our normal 20 day period. It was also in part because we launched our new mobile experience transitioning from a third party, Verve, to an entirely Drupal based mobile site. This shift greatly enhances the user's experience on a mobile device by leveraging block logic from the desktop site and providing access to all Examiner.com content. The Verve experience was limited to the previous 30 days.

read more

This is the link to the recording of chat with Dries about core development.

http://blip.tv/dries-buytaert/chat-with-dries-drupal-core-development-pr...

I figured it would be good to have this here.

Drupal 8 Initiatives

When I first started tackling the problem of https and Drupal caching I wanted an interim solution and started by making the database caching layer smarter. Neil Drumm had the idea to use protocol-relative (a.k.a, schemeless) URLs. This turns out to be a great idea and thanks to some hooks in Drupal it's something we can implement fairly easily.

Protocol-Relative URL

Protocol relative URLs don't contain the protocol. For example, http://example.com/foo/bar would be //example.com/foo/bar. Browsers will use the protocol for the page it's on when the protocol is absent. For a page that could be displayed under https or http this is a nice solution and can help us avoid caching for both situations.

Protocol-Relative URLs aren't new or novel. For example, Google has been using them for some time. Just take a look at the source of a Google Plus page (or many others).

The Setup

You first need to make sure your server is capable of serving both http and https requests. For URLs to not have a protocol it makes sense to serve with multiple protocols. There is also the case where your content could be served out of http on RSS then viewed in an RSS reader under https. You don't want problems to pop up in cases like this.

IE 7/8 and CSS

Yeah, so older versions of IE have a bug. IE 7 and 8 will download stylesheets twice when the protocol is missing. You can either code around this, not use this technique, or let those users suffer with slightly slower performance.

Implementation Options

What follows are two different implementation options depending on what you want to do and your scenario. You can pick the case you think works best for you or tweak to your hearts delight.

Images Only

Many of the images inside Drupal (e.g., those handled by image styles) have a full URLs. If you want to alter these it's possible through the use of hook_preprocess_image().

/**
 * Implementation of hook_preprocess_image().
 *
 * Make images that use a full url be protocol relative.
 */
function custom_preprocess_image(&$variables) {
 
  // If the image URL starts with a protocol remove it and use a
  // relative protocol.
  $scheme = file_uri_scheme($variables['path']);
  $protocols = array('http', 'https');
  if ($scheme && in_array($scheme, $protocols)) {
    $variables['path'] = '//' . file_uri_target($variables['path']);
  }
}

This code is available as a gist at https://gist.github.com/1523736

All File URLs Generated By Drupal

In Drupal file_create_url() is used to create the full URLs for everything from image to CSS and JavaScript. It handles public and private files, internal paths (which it turns into full URLs), and making sure everything is tidy. To accommodate CDNs there is hook_file_url_alter() which we can plug into in order to make protocol-relative URLs.

/**
 * Implements hook_file_url_alter().
 *
 * Make all URLs be protocol relative.
 * Note: protocol relatice URLs will cause IE7/8 to download stylesheets twice.
 */
function custom_file_url_alter(&$url) {
 
  global $base_url;
 
  static $relative_base_url = NULL, $relative_base_length = NULL;
 
  $scheme = file_uri_scheme($url);
 
  // For some things (e.g., images) hook_file_url_alter can be called multiple
  // times. So, we have to be sure not to alter it multiple times. If we already
  // are relative protocol we can just return.
  // Only setup the and parse this stuff once.
  if (!$relative_base_url || !$relative_base_length) {
    $relative_base_url = '//' . file_uri_target($base_url);
    $relative_base_length = strlen($relative_base_url);
  }
  if (!$scheme && substr($url, 0, $relative_base_length) == $relative_base_url) {
    return;
  }
 
  // Handle the case where we have public files with the scheme public:// or
  // the case the relative path doesn't start with a /. Internal relative urls
  // have the base url prepended to them.
  if (!$scheme || $scheme == 'public') {
 
    // Internal Drupal paths.
    if (!$scheme) {
      $path = $url;
    }
    else {
      $wrapper = file_stream_wrapper_get_instance_by_scheme($scheme);
      $path = $wrapper->getDirectoryPath() . '/' . file_uri_target($url);
    }
 
    // Clean up Windows paths.
    $path = str_replace('\\', '/', $path);
 
    $url = $base_url . '/' . $path;
  }
 
  // Convert full URLs to relative protocol.
  $protocols = array('http', 'https');
  $scheme = file_uri_scheme($url);
  if ($scheme && in_array($scheme, $protocols)) {
    $url = '//' . file_uri_target($url);
  }
}

This code is available as a gist at https://gist.github.com/1524135

Be careful with this option as it will cause CSS to be downloaded twice in IE 7 and 8.

In this blog we ll explain how to set up a geospatial search by radius using apache solr geospatial extension in a drupal 7.

You ll need the following modules to make the geospatial search by radius work:

Read moreSubmitted by Dominique De Cooman on Tue, 2011-12-27 10:38apachesolr geospatial local solr searchDrupal 7 local apache solr: Geospatial search by radius

read more

One of the problems I recently discovered with the Drupal cache is that it doesn't properly handle https transactions when caching happens. Let's take a look at understanding the problem, an interim solution for the database cache, and finding a long term solution.

The Problem

The page cache properly respects https because it uses a full url including the protocol when generating a cache id. But, Drupal uses multiple layers of caching with the html in a page. For example, the content of blocks can be cached. What if the html for a block has absolute URLs pointing back to the site and those are generated on the http version of the site. Then this is cached. Then this cache is used to create the https version of the page. Then those cached absolute URLs are not using https.

Media Module, Where I Found The Problem

The example that caused me to understand this problem was the media module. With the media module you can embed images and video into a text area (like the body of an article). Media tags are converted to html by the filter system and the results are cached. Images, for example, use absolute URLs (this is part of core and is a good thing for some use cases). If the cache for this text was generated on the http version of the site the path to the image on the https version of the page will use http as the protocol.

This opens up all kids of possible issues. Just imagine someone in a coffee shop thinking they are on https only to have cookies sent back to the server for that image in plain text. Or maybe you have a better imagination than me and can think of something more sinister.

Nested Caching

The issues is nested caching of html and how many of the nested caches don't respect the protocol like the page cache does. This can apply to any html that's cached across any modules.

Fixing Database Caching In The Short Term

To work around this problem I've started a database caching layer that works around this problem by respecting https for most caches. I don't really like this solution and there are some definite hacks in how it works. The real solution should be to have the code saving and retrieving from the html caches be smart enough to include the protocol in the caches. Unfortunately, a cache backend can't fix the places the cache is used.

A Long Term Solution

An issue for a long term solution is already open. If you have any experience in this area, it impacts your sites, or you want to help out please head over there so we can fix this moving forward.

While patching Views I added this to an existing JS file thanks to caseyn. You can replace the iframe-related code (which is just two lines) to add your JS to Drupal:

 

 

 

read more