Google AI Video Moderation

Cloudinary is a cloud-based service that provides an end-to-end image and video management solution including uploads, storage, manipulations, optimizations and delivery. Cloudinary offers a rich set of manipulation and analysis capabilities and allows you to upload images and videos to the cloud, manipulate them on-the-fly and deliver them to your users optimized and cached via a fast CDN.

Google Cloud Video Intelligence is a service that makes it easy to add video analysis to your applications. Cloudinary provides an add-on for Google's automatic video moderation service, fully integrated into Cloudinary's video management and manipulation pipeline.

With the Google AI Moderation add-on, you can extend Cloudinary's powerful cloud-based media library and delivery capabilities with automatic AI-based moderation of your videos. Protect your users from explicit and suggestive adult content in your user-uploaded videos, making sure that no offensive videos are displayed to your viewers.

Automatic video moderation flow

The following describes the flow of uploading and displaying moderated videos using Cloudinary and the Google AI Video Moderation add-on:

1. Your users upload a video to Cloudinary through your application.
2. The uploaded video is sent to Google for moderation.
3. The video is marked as either approved or rejected based on the results returned by Google.
4. An optional notification callback is sent to your application with the video moderation result.
5. A rejected video is moved to a secondary backup repository, consuming additional storage.
6. Moderated videos can be listed programmatically using Cloudinary's Admin API or interactively using the Media Library in your account console.
7. You can manually override the automatic moderation results using the Admin API or the Media Library.
   

Moderation categorization

Google assigns a moderation confidence level indicating the chances that a video contains unacceptable content. The likelihood is given as a value on the following scale: very_unlikely, unlikely, possible, likely, and very_likely.

The default moderation confidence level for rejecting an image is likely, unless specifically overridden (see explanation and examples below). All videos classified by Google with a value that is more severe than the rejection confidence level are classified as "rejected". Otherwise their status is set to "approved".

Request video moderation

To request moderation while uploading a video with the default rejection confidence level, set the moderation upload API parameter to google_video_moderation and set the resource_type parameter to video:

Cloudinary::Uploader.upload("my_file.mp4",
  :resource_type => "video", 
  :moderation => "google_video_moderation")

\Cloudinary\Uploader::upload("my_file.mp4", 
  array(
    "resource_type" => "video", 
    "moderation" => "google_video_moderation"));

cloudinary.uploader.upload("my_file.mp4",
  resource_type = "video", 
  moderation = "google_video_moderation")

cloudinary.v2.uploader.upload("my_file.mp4", 
  { resource_type: "video", 
    moderation: "google_video_moderation"},
  function(error, result){console.log(result);});

cloudinary.uploader().upload("my_file.mp4", 
  ObjectUtils.asMap(
    "resource_type", "video", 
    "moderation", "google_video_moderation"));

var uploadParams = new VideoUploadParams(){
  File = new FileDescription(@"my_file.mp4"),
  Moderation = "google_video_moderation"};
var uploadResult = cloudinary.Upload(uploadParams);

By default, if any frame in the video returns a result of at least likely, then the video will be rejected. You can override the default rejection confidence level of likely by including the new value as part of the 'moderation' parameter value, separated by a colon. Possible rejection values are: unlikely, possible, likely, and very_likely.

For example, to request moderation with the rejection confidence level set to possible:

Cloudinary::Uploader.upload("my_file.mp4", 
  :resource_type => "video", 
  :moderation => "google_video_moderation:possible")

\Cloudinary\Uploader::upload("my_file.mp4", 
  array(
    "resource_type" => "video", 
    "moderation" => "google_video_moderation:possible"));

cloudinary.uploader.upload("my_file.mp4",
  resource_type = "video", 
  moderation = "google_video_moderation:possible")

cloudinary.v2.uploader.upload("my_file.mp4", 
  { resource_type: "video", 
    moderation: "google_video_moderation:possible"},
  function(error, result){console.log(result);});

cloudinary.uploader().upload("my_file.mp4", 
  ObjectUtils.asMap(
    "resource_type", "video", 
    "moderation", "google_video_moderation:possible"));

var uploadParams = new VideoUploadParams(){
  File = new FileDescription(@"my_file.mp4"),
  Moderation = "google_video_moderation:possible"};
var uploadResult = cloudinary.Upload(uploadParams);

Moderation response

The following snippet shows an example of a response to an upload API call, indicating the results of the moderation request, including the individual frames that failed the moderation evaluation, and shows that the video has been placed in 'rejected' moderation status.

{
...
"moderation_response":
  {"moderation_confidence": "POSSIBLE",
   "frames":
     [{"pornography_likelihood": "POSSIBLE",
       "time_offset": 0.510936},
      {"pornography_likelihood": "LIKELY",
      "time_offset": 2.832345},
      ...
      {"pornography_likelihood": "VERY_LIKELY",
      "time_offset": 14.014693}],
   "moderation_status": "rejected",
   "moderation_kind": "google_video_moderation",
   "moderation_updated_at": "2018-11-13T12:37:30Z"}
 ...
}

Video moderation listing

Cloudinary's Admin API can be used to list all moderated videos. You can list either the approved or the rejected videos by specifying the second parameter of the resources_by_moderation API method. For example, to list all videos rejected by the Google AI Video Moderation add-on:

Cloudinary::Api.resources_by_moderation("google_video_moderation", "rejected")

$api->resources_by_moderation("google_video_moderation", "rejected");

cloudinary.api.resources_by_moderation("google_video_moderation", "rejected")

cloudinary.api.resources_by_moderation(google_video_moderation, 'rejected', 
  function(result)  { console.log(result); });

cloudinary.api().resourcesByModeration("google_video_moderation", "rejected", 
  ObjectUtils.emptyMap());

var listResourcesResult = cloudinary.ListResourcesByModerationStatus("google_video_moderation", "rejected");

{
 "resources": 
  [{
    "public_id": "q7vcvrfjm9mj4bfp3qc8",
    "format": "mp4",
    "version": 1393794403,
    "resource_type": "video",
    "type": "upload",
    "created_at": "2017-05-04T41:02:23Z",
    "bytes": 120253,
    "width": 864,
    "height": 576,
    "backup": true,
    "url": "http://res.cloudinary.com/demo/video/upload/v1393794403/q7vcvrfjm9mj4bfp3qc8.mp4",
    "secure_url": "https://res.cloudinary.com/demo/video/upload/v1393794403/q7vcvrfjm9mj4bfp3qc8.mp4"
  },
  {
    "public_id": "zp4fgdbabhlwwa7bxu84",
    "format": "mp4",
    ...
  }
  ...
 ]
}

Manual override

As the automatic video moderation of the Google AI Moderation add-on is based on a decision made by an advanced artificial intelligence algorithm, in some cases you may want to manually override the automatic moderation decision and either approve a previously rejected video or reject an approved one.

Overriding moderation via the Media Library

One way to manually override the moderation result is using Cloudinary's Media Library Web interface. From the left navigation menu, select Moderation. Then, from moderation tools list in the top menu, select Google Video and then select which moderation queue of videos to display (Pending, Rejected, or Approved).

• When listing the queue of videos rejected by Google, you can click on the green Approve button to revert the decision and recover the original rejected video.
• When listing the queue of videos approved by Google, you can click on the red Reject button to revert the decision and prevent a certain video from being publicly available to your users.
  

Overriding moderation via the Admin API

Alternatively to using the Media Library interface, you can use Cloudinary's Admin API to manually override the moderation result. The following example uses the update API method while specifying a public ID of a moderated video and setting the moderation_status parameter to the rejected status.

Cloudinary::Api.update("hwepb67oxzh4lrigssld", 
  :resource_type => "video", 
  :moderation_status => "rejected")

$api->update("hwepb67oxzh4lrigssld", 
  array(
    "resource_type" => "video", 
    "moderation_status" => "rejected"));

cloudinary.api.update("hwepb67oxzh4lrigssld",
  resource_type = "video", 
  moderation_status = "rejected")

cloudinary.v2.api.update("hwepb67oxzh4lrigssld",
  { resource_type: "video", 
    moderation_status: "rejected" },
  function(error, result){console.log(result);});

cloudinary.api().update("hwepb67oxzh4lrigssld", 
  ObjectUtils.asMap(
    "resource_type", "video", 
    "moderation_status", "rejected"));

var updateParams = new UpdateParams("hwepb67oxzh4lrigssld"){
  ResourceType = "video",
  ModerationStatus = "rejected"};
var updateResult = cloudinary.Update(updateParams);