Advanced Facial Attributes Detection
Overview
Cloudinary provides an add-on for advanced face attribute detection capabilities, fully integrated into Cloudinary's image management and manipulation pipeline. With the Advanced Facial Attribute Detection add-on, you can extend the Cloudinary features that involve semantic photo data extraction, image cropping and the positioning of image overlays. When using the Advanced Facial Attribute Detection add-on, your images are further processed and additional advanced face attributes are automatically extracted. Cloudinary can then use these additional details to smartly crop, position, rotate and overlay images.
Cloudinary is a cloud-based service that provides an end-to-end image management solution including uploads, storage, manipulations, optimizations and delivery, and offers a rich set of image management, manipulation, analysis and cropping capabilities.
The Advanced Facial Attribute Detection add-on is an integrated face detection solution that utilizes Project Oxford developed by Microsoft. Project Oxford provides high precision face location detection with state-of-the-art cloud-based algorithms that can detect up to 64 human faces in an image. The detected faces are returned with rectangles (left, top, width and height) indicating the location of faces in the image in pixels, the exact position details of the eyes, mouth, eyebrows, nose and lips, as well as a series of face related attributes from each face such as pose, gender and age.
Face attribute detection on upload
The Advanced Facial Attribute Detection add-on is very simple to use. The extensive list of face attributes can be extracted from any photo when calling Cloudinary's upload API and setting the detection parameter to adv_face. For example, take a look at the following photo that we want to upload to Cloudinary's demo account:
When calling Cloudinary's API to upload the photo, the detection parameter is added to extract the face attributes in the uploaded photo:
Cloudinary::Uploader.upload("coupled.jpg", :detection => "adv_face")
\Cloudinary\Uploader::upload("coupled.jpg", array("detection" => "adv_face"));
cloudinary.uploader.upload("coupled.jpg", detection = "adv_face")
cloudinary.uploader.upload("coupled.jpg", function(result) { console.log(result); }, { detection: "adv_face" });
cloudinary.uploader().upload("coupled.jpg", ObjectUtils.asMap("detection", "adv_face"));
The JSON snippet below contains the results of applying automatic face attribute detection on the uploaded image. The response includes very detailed information regarding the two faces that were automatically detected in the example photo:
bounding_box- The bounding box surrounding each detected face.head_pose- The way the face is positioned in 3D.facial_landmarks- Exact position details of the eyes, mouth, eyebrows, nose and lips.gender- Whether the person is a male or a female.age- Estimated age of the person.
{
...
"info"=>
{"detection"=>
{"adv_face"=>
{"status"=>"complete",
"data"=>
[{"bounding_box"=>
{"top"=>96.0, "left"=>106.0, "width"=>47.0, "height"=>47.0},
"attributes"=>
{"head_pose"=>{"pitch"=>0.0, "roll"=>-9.6, "yaw"=>11.4},
"gender"=>"male",
"age"=>60},
"facial_landmarks"=>
{"mouth"=>
{"left"=>{"x"=>119.0, "y"=>131.9},
"right"=>{"x"=>140.5, "y"=>128.9},
"under_lip"=>
{"bottom"=>{"x"=>131.3, "y"=>134.3},
"top"=>{"x"=>131.3, "y"=>132.4}},
"upper_lip"=>
{"bottom"=>{"x"=>130.7, "y"=>128.9},
"top"=>{"x"=>130.7, "y"=>127.5}}},
"eyebrow"=>
{"left_outer"=>{"x"=>108.9, "y"=>113.0},
"left_inner"=>{"x"=>122.3, "y"=>106.7},
"right_inner"=>{"x"=>131.7, "y"=>105.6},
"right_outer"=>{"x"=>142.5, "y"=>108.4}},
"eye"=>
{"left_outer"=>{"x"=>113.7, "y"=>112.6},
"left_top"=>{"x"=>116.3, "y"=>110.7},
"left_bottom"=>{"x"=>116.8, "y"=>112.1},
"left_inner"=>{"x"=>118.8, "y"=>110.9},
"right_inner"=>{"x"=>134.1, "y"=>109.0},
"right_top"=>{"x"=>136.4, "y"=>107.8},
"right_bottom"=>{"x"=>136.5, "y"=>109.6},
"right_outer"=>{"x"=>139.0, "y"=>109.5},
"left_pupil"=>{"x"=>117.6, "y"=>111.1},
"right_pupil"=>{"x"=>137.2, "y"=>108.7}},
"nose"=>
{"tip"=>{"x"=>130.0, "y"=>119.2},
"root_left"=>{"x"=>124.7, "y"=>110.6},
"root_right"=>{"x"=>130.3, "y"=>109.9},
"left_alar_top"=>{"x"=>123.6, "y"=>116.4},
"right_alar_top"=>{"x"=>133.2, "y"=>115.2},
"left_alar_out_tip"=>{"x"=>121.3, "y"=>121.2},
"right_alar_out_tip"=>{"x"=>136.2, "y"=>119.0}}}},
{"bounding_box"=>
{"top"=>148.0, "left"=>144.0, "width"=>44.0, "height"=>44.0},
"attributes"=>
{"head_pose"=>{"pitch"=>0.0, "roll"=>-11.0, "yaw"=>-15.8},
"gender"=>"female",
"age"=>38},
"facial_landmarks"=>
{"mouth"=>
{"left"=>{"x"=>158.2, "y"=>181.2},
"right"=>{"x"=>178.9, "y"=>175.7},
"under_lip"=>
{"bottom"=>{"x"=>168.2, "y"=>184.6},
"top"=>{"x"=>167.5, "y"=>182.3}},
"upper_lip"=>
{"bottom"=>{"x"=>165.9, "y"=>177.0},
"top"=>{"x"=>165.1, "y"=>174.9}}},
"eyebrow"=>
{"left_outer"=>{"x"=>148.3, "y"=>160.7},
"left_inner"=>{"x"=>158.9, "y"=>158.3},
"right_inner"=>{"x"=>166.3, "y"=>155.3},
"right_outer"=>{"x"=>180.7, "y"=>154.4}},
"eye"=>
{"left_outer"=>{"x"=>152.0, "y"=>164.9},
"left_top"=>{"x"=>154.5, "y"=>162.6},
"left_bottom"=>{"x"=>155.1, "y"=>164.0},
"left_inner"=>{"x"=>157.7, "y"=>163.0},
"right_inner"=>{"x"=>171.3, "y"=>159.2},
"right_top"=>{"x"=>173.9, "y"=>157.6},
"right_bottom"=>{"x"=>174.0, "y"=>159.1},
"right_outer"=>{"x"=>176.7, "y"=>158.3},
"left_pupil"=>{"x"=>155.6, "y"=>163.2},
"right_pupil"=>{"x"=>174.9, "y"=>158.2}},
"nose"=>
{"tip"=>{"x"=>162.7, "y"=>169.5},
"root_left"=>{"x"=>160.3, "y"=>162.3},
"root_right"=>{"x"=>166.4, "y"=>160.7},
"left_alar_top"=>{"x"=>159.6, "y"=>167.4},
"right_alar_top"=>{"x"=>168.2, "y"=>165.7},
"left_alar_out_tip"=>{"x"=>157.9, "y"=>172.1},
"right_alar_out_tip"=>{"x"=>171.0, "y"=>168.1}}
}
}]
}
}
}Face attribute detection after uploading
The example above shows how to automatically detect face attributes of photos during their upload process. Alternatively, you can use Cloudinary's Admin API to apply automatic face attribute detection to already uploaded images, based on their public IDs. The following code sample uses Cloudinary's update API to apply face attribute detection to the uploaded image that has the 'coupled' public ID.
Cloudinary::Api.update("coupled", :detection => "adv_face")
\Cloudinary\Api::update("coupled", array("detection" => "adv_face"));
cloudinary.api.update("coupled", detection = "adv_face")
cloudinary.api.update("coupled", function(result) { console.log(result); }, { detection: "adv_face" });
cloudinary.api().update("coupled", ObjectUtils.asMap("detection", "adv_face"));
The face attributes that were previously extracted either using the 'upload' or 'update' API, are also available using the Admin API's show resource details method:
Cloudinary::Api.resource('coupled')
$api->resource("coupled");
cloudinary.api.resource("coupled")cloudinary.api.resource('coupled', function(result) { console.log(result) });
cloudinary.api().resource("coupled", ObjectUtils.emptyMap());Face detection based cropping
Cloudinary provides a large set of image manipulation and cropping options. Based on the position of facial attributes detected by the Advanced Facial Attribute Detection add-on, Cloudinary can crop your images to focus on the detected faces, while providing a large set of image transformation and cropping options when using a Cloudinary delivery URL or calling Cloudinary's image API.
To focus an automatic crop on the detected faces, simply set the crop parameter to thumb, fill or crop and the gravity parameter to adv_faces (set gravity to adv_face for focusing on the single largest detected face in the image).
The following code sample creates a 150x150 thumbnail of the coupled image that is generated using face-detection based cropping. The gravity parameter is set to adv_faces while the crop mode is set to thumb.
cl_image_tag("coupled.jpg", :gravity=>"adv_faces", :width=>150, :height=>150, :crop=>"thumb")
cl_image_tag("coupled.jpg", array("gravity"=>"adv_faces", "width"=>150, "height"=>150, "crop"=>"thumb"))
CloudinaryImage("coupled.jpg").image(gravity="adv_faces", width=150, height=150, crop="thumb")
cloudinary.image("coupled.jpg", {gravity: "adv_faces", width: 150, height: 150, crop: "thumb"})
cloudinary.url().transformation(new Transformation().gravity("adv_faces").width(150).height(150).crop("thumb")).imageTag("coupled.jpg")
$.cloudinary.image("coupled.jpg", {gravity: "adv_faces", width: 150, height: 150, crop: "thumb"})
cloudinary.Api.UrlImgUp.Transform(new Transformation().Gravity("adv_faces").Width(150).Height(150).Crop("thumb")).BuildImageTag("coupled.jpg")
Alternatively, you can crop an image based on the single most visible face in a photo. The following code sample generates and delivers a 100x100 thumbnail containing a single face. The gravity parameter is set to adv_face while the crop mode is set to thumb.
cl_image_tag("coupled.jpg", :gravity=>"adv_face", :width=>100, :height=>100, :crop=>"thumb")
cl_image_tag("coupled.jpg", array("gravity"=>"adv_face", "width"=>100, "height"=>100, "crop"=>"thumb"))
CloudinaryImage("coupled.jpg").image(gravity="adv_face", width=100, height=100, crop="thumb")
cloudinary.image("coupled.jpg", {gravity: "adv_face", width: 100, height: 100, crop: "thumb"})
cloudinary.url().transformation(new Transformation().gravity("adv_face").width(100).height(100).crop("thumb")).imageTag("coupled.jpg")
$.cloudinary.image("coupled.jpg", {gravity: "adv_face", width: 100, height: 100, crop: "thumb"})
cloudinary.Api.UrlImgUp.Transform(new Transformation().Gravity("adv_face").Width(100).Height(100).Crop("thumb")).BuildImageTag("coupled.jpg")
Face detection with signed URLs
Cloudinary's dynamic image manipulation URLs are quite powerful, however, due to the potential costs of users accessing unplanned dynamic URLs with the Advanced Facial Attribute Detection cropping directives, we request that you sign your image manipulation URLs using Cloudinary's authenticated API and with the sign_url parameter set to true.
Note you can create dynamic unsigned image manipulation URLs based on Advanced Facial Attribute Detection as long as you first extract the face attributes during upload or by using the Admin API.
For example, the following code snippet generates a signed URL that can be used when the facial attributes have not been previously extracted:
cloudinary_url("coupled.jpg", :crop => "thumb", :width => 100, :height => 100, :gravity => "adv_face", :sign_url => true)
cloudinary_url("coupled.jpg", array("crop" => "thumb", "width" => 100, "height" => 100, "gravity" => "adv_face", "sign_url" => true));
cloudinary.CloudinaryImage("coupled.jpg").build_url( crop = "thumb", width = 100, height = 100, gravity = "adv_face", sign_url = True)
cloudinary.url("coupled.jpg", { crop: "thumb", width: 100, height: 100, gravity: "adv_face", sign_url: true });
cloudinary.url().transformation(
new Transformation().crop("thumb").width(100).height(100).
gravity("adv_face")).
signed(true).generate("coupled.jpg");The code above dynamically generates the following signed URL:
http://res.cloudinary.com/demo/image/upload/s--vnhlc4WH--/c_thumb,g_adv_face,h_100,w_100/coupled.jpg
As you can see, the generated Cloudinary URL includes a signature component (s--vnhlc4WH--). Only URLs with a valid signature will be approved for on-the-fly image manipulation using advanced facial attribute detection.
Eyes detection based cropping
In addition to face-detection based cropping, Cloudinary can dynamically crop your images based on the position of detected eyes. Simply set the gravity parameter to adv_eyes (g_adv_eyes for URLs) to center the image on the detected eyes. The example below delivers a 100x40 thumbnail centered on the eyes of the woman in the photo.
cl_image_tag("face1.jpg", :width=>100, :height=>40, :gravity=>"adv_eyes", :crop=>"thumb")
cl_image_tag("face1.jpg", array("width"=>100, "height"=>40, "gravity"=>"adv_eyes", "crop"=>"thumb"))
CloudinaryImage("face1.jpg").image(width=100, height=40, gravity="adv_eyes", crop="thumb")
cloudinary.image("face1.jpg", {width: 100, height: 40, gravity: "adv_eyes", crop: "thumb"})
cloudinary.url().transformation(new Transformation().width(100).height(40).gravity("adv_eyes").crop("thumb")).imageTag("face1.jpg")
$.cloudinary.image("face1.jpg", {width: 100, height: 40, gravity: "adv_eyes", crop: "thumb"})
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(100).Height(40).Gravity("adv_eyes").Crop("thumb")).BuildImageTag("face1.jpg")
Eyes detection for accurate red eye removal
Cloudinary's rich manipulation capabilities already allow you to automate red eye removal by setting the effect parameter to redeye (e_redeye for URLs). This enables smart red eye removal algorithms to be automatically applied on-the-fly to uploaded images.
In order to get even better quality results, you can use Cloudinary’s Advanced Facial Attribute Detection add-on for eye detection together with the red eye removal effect. The add-on can automatically detect where eyes are located in a photo and then the red eye removal algorithm can be applied in a more precise way: simply set the effect parameter to adv_redeye (e_adv_redeye for URLs).
For example, an image named redeye was uploaded to Cloudinary. A cropped 300x80 thumbnail of the photo centered on the eyes only for illustration is displayed below:
cl_image_tag("redeye.jpg", :width=>300, :height=>80, :gravity=>"adv_eyes", :crop=>"thumb")
cl_image_tag("redeye.jpg", array("width"=>300, "height"=>80, "gravity"=>"adv_eyes", "crop"=>"thumb"))
CloudinaryImage("redeye.jpg").image(width=300, height=80, gravity="adv_eyes", crop="thumb")
cloudinary.image("redeye.jpg", {width: 300, height: 80, gravity: "adv_eyes", crop: "thumb"})
cloudinary.url().transformation(new Transformation().width(300).height(80).gravity("adv_eyes").crop("thumb")).imageTag("redeye.jpg")
$.cloudinary.image("redeye.jpg", {width: 300, height: 80, gravity: "adv_eyes", crop: "thumb"})
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(300).Height(80).Gravity("adv_eyes").Crop("thumb")).BuildImageTag("redeye.jpg")
Adding the redeye effect parameter and setting its value to adv_redeye delivers the following image:
cl_image_tag("redeye.jpg", :transformation=>[ {:effect=>"adv_redeye"}, {:width=>300, :height=>80, :gravity=>"adv_eyes", :crop=>"thumb"} ])
cl_image_tag("redeye.jpg", array("transformation"=>array( array("effect"=>"adv_redeye"), array("width"=>300, "height"=>80, "gravity"=>"adv_eyes", "crop"=>"thumb") )))
CloudinaryImage("redeye.jpg").image(transformation=[ {"effect": "adv_redeye"}, {"width": 300, "height": 80, "gravity": "adv_eyes", "crop": "thumb"} ])
cloudinary.image("redeye.jpg", {transformation: [ {effect: "adv_redeye"}, {width: 300, height: 80, gravity: "adv_eyes", crop: "thumb"} ]})
cloudinary.url().transformation(new Transformation() .effect("adv_redeye").chain() .width(300).height(80).gravity("adv_eyes").crop("thumb")).imageTag("redeye.jpg")
$.cloudinary.image("redeye.jpg", {transformation: [ {effect: "adv_redeye"}, {width: 300, height: 80, gravity: "adv_eyes", crop: "thumb"} ]})
cloudinary.Api.UrlImgUp.Transform(new Transformation()
.Effect("adv_redeye").Chain()
.Width(300).Height(80).Gravity("adv_eyes").Crop("thumb")).BuildImageTag("redeye.jpg")
Face overlays
The extracted facial attribute details also include the pose of the detected face, which allows Cloudinary to not only position overlays on top of detected faces, but also to rotate and scale the overlays accordingly for exact positioning over the underlying face. This is easily accomplished by setting the gravity parameter of the added overlay to adv_faces.
For example, the following PNG image named silver_face_mask was uploaded to Cloudinary:
To smartly overlay the mask image on top of all detected faces in the coupled image, the overlay parameter is set to the ID of the mask image and the gravity parameter is set to adv_faces. In addition, we set the region_relative flag together with a 1.1 width, which means that each overlay will be scaled to 110% of the width of the detected face.
cl_image_tag("coupled.jpg", :flags=>"region_relative", :gravity=>"adv_faces", :overlay=>"silver_face_mask", :width=>1.1)
cl_image_tag("coupled.jpg", array("flags"=>"region_relative", "gravity"=>"adv_faces", "overlay"=>"silver_face_mask", "width"=>1.1))
CloudinaryImage("coupled.jpg").image(flags="region_relative", gravity="adv_faces", overlay="silver_face_mask", width=1.1)
cloudinary.image("coupled.jpg", {flags: "region_relative", gravity: "adv_faces", overlay: "silver_face_mask", width: 1.1})
cloudinary.url().transformation(new Transformation().flags("region_relative").gravity("adv_faces").overlay("silver_face_mask").width(1.1)).imageTag("coupled.jpg")
$.cloudinary.image("coupled.jpg", {flags: "region_relative", gravity: "adv_faces", overlay: "silver_face_mask", width: 1.1})
cloudinary.Api.UrlImgUp.Transform(new Transformation().Flags("region_relative").Gravity("adv_faces").Overlay("silver_face_mask").Width(1.1)).BuildImageTag("coupled.jpg")
Eyes overlays
As described above, the Advanced Facial Attribute Detection add-on detects specific facial attributes, including the exact position of the eyes of each face in a photo. Based on this information, Cloudinary can position overlays on top of all the detected eye pairs in an image.
For example, the following PNG image named glasses was uploaded to Cloudinary:
In order to smartly overlay the glasses on top of all detected eye pairs in an uploaded image, the overlay parameter is set to the ID of the glasses image and the gravity parameter is set to adv_eyes. In addition, we set the region_relative flag together with a 1.5 width, which means that each overlay should be scaled to 150% of the detected eyes.
cl_image_tag("coupled.jpg", :flags=>"region_relative", :gravity=>"adv_eyes", :overlay=>"glasses", :width=>1.5)
cl_image_tag("coupled.jpg", array("flags"=>"region_relative", "gravity"=>"adv_eyes", "overlay"=>"glasses", "width"=>1.5))
CloudinaryImage("coupled.jpg").image(flags="region_relative", gravity="adv_eyes", overlay="glasses", width=1.5)
cloudinary.image("coupled.jpg", {flags: "region_relative", gravity: "adv_eyes", overlay: "glasses", width: 1.5})
cloudinary.url().transformation(new Transformation().flags("region_relative").gravity("adv_eyes").overlay("glasses").width(1.5)).imageTag("coupled.jpg")
$.cloudinary.image("coupled.jpg", {flags: "region_relative", gravity: "adv_eyes", overlay: "glasses", width: 1.5})
cloudinary.Api.UrlImgUp.Transform(new Transformation().Flags("region_relative").Gravity("adv_eyes").Overlay("glasses").Width(1.5)).BuildImageTag("coupled.jpg")
In another example, the following PNG image named harlequinmask was uploaded to Cloudinary:
To smartly overlay the mask on top of all the detected eye pairs in the uploaded image cloudinary_team, the overlay parameter is set to the ID of the harlequinmask image and the gravity parameter is set to adv_eyes. We also set the region_relative flag together with a 1.7 width to scale the overlay to 170% of the width of the detected eyes, and resize the image to an oval thumbnail with a width of 700 pixels.
cl_image_tag("cloudinary_team.jpg", :transformation=>[ {:width=>700, :radius=>"max"}, {:flags=>"region_relative", :gravity=>"adv_eyes", :overlay=>"harlequinmask", :width=>1.7} ])
cl_image_tag("cloudinary_team.jpg", array("transformation"=>array( array("width"=>700, "radius"=>"max"), array("flags"=>"region_relative", "gravity"=>"adv_eyes", "overlay"=>"harlequinmask", "width"=>1.7) )))
CloudinaryImage("cloudinary_team.jpg").image(transformation=[ {"width": 700, "radius": "max"}, {"flags": "region_relative", "gravity": "adv_eyes", "overlay": "harlequinmask", "width": 1.7} ])
cloudinary.image("cloudinary_team.jpg", {transformation: [ {width: 700, radius: "max"}, {flags: "region_relative", gravity: "adv_eyes", overlay: "harlequinmask", width: 1.7} ]})
cloudinary.url().transformation(new Transformation() .width(700).radius("max").chain() .flags("region_relative").gravity("adv_eyes").overlay("harlequinmask").width(1.7)).imageTag("cloudinary_team.jpg")
$.cloudinary.image("cloudinary_team.jpg", {transformation: [ {width: 700, radius: "max"}, {flags: "region_relative", gravity: "adv_eyes", overlay: "harlequinmask", width: 1.7} ]})
cloudinary.Api.UrlImgUp.Transform(new Transformation()
.Width(700).Radius("max").Chain()
.Flags("region_relative").Gravity("adv_eyes").Overlay("harlequinmask").Width(1.7)).BuildImageTag("cloudinary_team.jpg")
For a full list of additional Cloudinary's image manipulation options, see Image transformations.