Google Research Blog
The latest news from Research at Google
The Question of Quantum Supremacy
Friday, May 04, 2018
Posted by Sergio Boixo, Research Scientist and Theory Team Lead, and Charles Neill, Quantum Electronics Engineer, Quantum A.I. Lab
Quantum computing
integrates the two largest technological revolutions of the last half century,
information technology
and
quantum mechanics
. If we compute using the rules of quantum mechanics, instead of binary logic, some
intractable
computational tasks
become feasible. An important goal in the pursuit of a universal quantum computer is the determination of the smallest computational task that is prohibitively hard for today’s classical computers. This crossover point is known as the “quantum supremacy” frontier, and is a critical step on the path to more powerful and useful computations.
In “
Characterizing quantum supremacy in near-term devices
” published in Nature Physics (arXiv
here
), we present the theoretical foundation for a practical demonstration of quantum supremacy in near-term devices. It proposes the task of sampling bit-strings from the output of random quantum circuits, which can be thought of as the “hello world” program for quantum computers. The upshot of the argument is that the output of random chaotic systems (think
butterfly effect
) become very quickly harder to predict the longer they run. If one makes a random, chaotic
qubit
system and examines how long a classical system would take to emulate it, one gets a good measure of when a quantum computer could outperform a classical one.
Arguably
, this is the strongest theoretical proposal to prove an
exponential
separation
between the
computational
power of classical and quantum computers.
Determining where exactly the quantum supremacy frontier lies for sampling random quantum circuits has rapidly become an exciting area of research. On one hand,
improvements
in
classical
algorithms
to
simulate quantum circuits
aim to
increase the size
of the quantum circuits required to establish quantum supremacy. This forces an experimental quantum device with a sufficiently large number of qubits and low enough error rates to implement circuits of sufficient depth (i.e the number of layers of gates in the circuit) to achieve supremacy. On the other hand, we now understand better how the particular choice of the quantum gates used to build random quantum circuits affects the
simulation cost
, leading to improved benchmarks for near-term quantum supremacy (available for download
here
), which are in some cases quadratically more expensive to simulate classically than the original proposal.
Sampling from random quantum circuits is an excellent calibration benchmark for quantum computers, which we call
cross-entropy benchmarking
. A successful quantum supremacy experiment with random circuits would demonstrate the basic building blocks for a large-scale
fault-tolerant
quantum computer. Furthermore, quantum physics has not yet been tested for
highly complex
quantum states such as this.
Space-time volume
of a quantum circuit computation. The computational cost for quantum simulation increases with the volume of the quantum circuit, and in general grows exponentially with the number of qubits and the circuit depth. For asymmetric grids of qubits, the computational space-time volume grows slower with depth than for symmetric grids, and can result in circuits exponentially
easier to simulate
.
In “
A blueprint for demonstrating quantum supremacy with superconducting qubits
” (arXiv
here
), we illustrate a blueprint towards quantum supremacy and experimentally demonstrate a proof-of-principle version for the first time. In the paper, we discuss two key ingredients for quantum supremacy: exponential complexity and accurate computations. We start by running algorithms on subsections of the device ranging from 5 to 9 qubits. We find that the classical simulation cost grows exponentially with the number of qubits. These results are intended to provide a clear example of the exponential power of these devices. Next, we use
cross-entropy benchmarking
to compare our results against that of an ordinary computer and show that our computations are highly accurate. In fact, the error rate is low enough to achieve quantum supremacy with a larger quantum processor.
Beyond achieving quantum supremacy, a quantum platform should offer clear applications. In our paper, we apply our algorithms towards computational problems in quantum statistical-mechanics using complex multi-qubit gates (as opposed to the two-qubit gates designed for a
digital quantum processor
with
surface code error correction
). We show that our devices can be used to study fundamental properties of materials, e.g. microscopic differences between metals and insulators. By extending these results to next-generation devices with ~50 qubits, we hope to answer scientific questions that are beyond the capabilities of any other computing platform.
Photograph of two gmon superconducting qubits and their tunable coupler developed by Charles Neill and Pedram Roushan.
These two publications introduce a realistic proposal for near-term quantum supremacy, and demonstrate a proof-of-principle version for the first time. We will continue to decrease the error rates and increase the number of qubits in
quantum processors
to reach the quantum supremacy frontier, and to develop
quantum algorithms
for useful near-term
applications
.
Google at ICLR 2018
Sunday, April 29, 2018
Posted by Jeff Dean, Google Senior Fellow, Head of Google Research and Machine Intelligence
This week, Vancouver, Canada hosts the
6th International Conference on Learning Representations
(ICLR 2018), a conference focused on how one can learn meaningful and useful representations of data for
machine learning
. ICLR includes conference and workshop tracks, with invited talks along with oral and poster presentations of some of the latest research on deep learning, metric learning, kernel learning, compositional models, non-linear structured prediction, and issues regarding non-convex optimization.
At the forefront of innovation in cutting-edge technology in
neural networks
and
deep learning
, Google focuses on both theory and application, developing learning approaches to understand and generalize. As Platinum Sponsor of ICLR 2018, Google will have a strong presence with over 130 researchers attending, contributing to and learning from the broader academic research community by presenting papers and posters, in addition to participating on organizing committees and in workshops.
If you are attending ICLR 2018, we hope you'll stop by our booth and chat with our researchers about the projects and opportunities at Google that go into solving interesting problems for billions of people. You can also learn more about our research being presented at ICLR 2018 in the list below (Googlers highlighted in
blue
)
Senior Program Chair:
Tara Sainath
Steering Committee includes:
Hugo Larochelle
Oral Contributions
Wasserstein Auto-Encoders
Ilya Tolstikhin,
Olivier Bousquet
,
Sylvain Gelly
, Bernhard Scholkopf
On the Convergence of Adam and Beyond
(Best Paper Award)
Sashank J. Reddi
,
Satyen Kale
,
Sanjiv Kumar
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
Christian Buck
,
Jannis Bulian
,
Massimiliano Ciaramita
,
Wojciech Gajewski
,
Andrea Gesmundo
,
Neil Houlsby
,
Wei Wang
Beyond Word Importance: Contextual Decompositions to Extract Interactions from LSTMs
W. James Murdoch,
Peter J. Liu
, Bin Yu
Conference Posters
Boosting the Actor with Dual Critic
Bo Dai, Albert Shaw, Niao He,
Lihong Li
, Le Song
MaskGAN: Better Text Generation via Filling in the _______
William Fedus
,
Ian Goodfellow
,
Andrew M. Dai
Scalable Private Learning with PATE
Nicolas Papernot, Shuang Song,
Ilya Mironov
,
Ananth Raghunathan
,
Kunal Talwar
,
Ulfar Erlingsson
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Yujun Lin,
Song Han
, Huizi Mao, Yu Wang, William J. Dally
Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
Yeming Wen, Paul Vicol, Jimmy Ba,
Dustin Tran
, Roger Grosse
Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models
Adam Roberts
,
Jesse Engel
,
Matt Hoffman
Multi-Mention Learning for Reading Comprehension with Neural Cascades
Swabha Swayamdipta,
Ankur P. Parikh
,
Tom Kwiatkowski
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
Adams Wei Yu,
David Dohan
,
Thang Luong
,
Rui Zhao
,
Kai Chen
,
Mohammad Norouzi
,
Quoc V. Le
Sensitivity and Generalization in Neural Networks: An Empirical Study
Roman Novak
,
Yasaman Bahri
,
Daniel A. Abolafia
,
Jeffrey Pennington
,
Jascha Sohl-Dickstein
Action-dependent Control Variates for Policy Optimization via Stein Identity
Hao Liu, Yihao Feng, Yi Mao,
Dengyong Zhou
, Jian Peng, Qiang Liu
An Efficient Framework for Learning Sentence Representations
Lajanugen Logeswaran,
Honglak Lee
Fidelity-Weighted Learning
Mostafa Dehghani, Arash Mehrjou,
Stephan Gouws
, Jaap Kamps, Bernhard Sch繹lkopf
Generating Wikipedia by Summarizing Long Sequences
Peter J. Liu
,
Mohammad Saleh
,
Etienne Pot
,
Ben Goodrich
,
Ryan Sepassi
,
Lukasz Kaiser
,
Noam Shazeer
Matrix Capsules with EM Routing
Geoffrey Hinton
,
Sara Sabour
,
Nicholas Frosst
Temporal Difference Models: Model-Free Deep RL for Model-Based Control
Sergey Levine,
Shixiang Gu
, Murtaza Dalal, Vitchyr Pong
Deep Neural Networks as Gaussian Processes
Jaehoon Lee
,
Yasaman Bahri
,
Roman Novak
,
Samuel L. Schoenholz
,
Jeffrey Pennington
,
Jascha Sohl-Dickstein
Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence at Every Step
William Fedus
, Mihaela Rosca, Balaji Lakshminarayanan,
Andrew M. Dai
, Shakir Mohamed,
Ian Goodfellow
Initialization Matters: Orthogonal Predictive State Recurrent Neural Networks
Krzysztof Choromanski
, Carlton Downey, Byron Boots
Learning Differentially Private Recurrent Language Models
H. Brendan McMahan
,
Daniel Ramage
,
Kunal Talwar
,
Li Zhang
Learning Latent Permutations with Gumbel-Sinkhorn Networks
Gonzalo Mena,
David Belanger
, Scott Linderman,
Jasper Snoek
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
Benjamin Eysenbach
,
Shixiang Gu
,
Julian Ibarz
,
Sergey Levine
Meta-Learning for Semi-Supervised Few-Shot Classification
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell,
Kevin Swersky
, Josh Tenenbaum,
Hugo Larochelle
, Richard Zemel
Thermometer Encoding: One Hot Way to Resist Adversarial Examples
Jacob Buckman
,
Aurko Roy
,
Colin Raffel
,
Ian Goodfellow
A Hierarchical Model for Device Placement
Azalia Mirhoseini
,
Anna Goldie
,
Hieu Pham
,
Benoit Steiner
,
Quoc V. Le
,
Jeff Dean
Monotonic Chunkwise Attention
Chung-Cheng Chiu
,
Colin Raffel
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
Kimin Lee,
Honglak Lee
, Kibok Lee, Jinwoo Shin
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
Ofir Nachum
,
Mohammad Norouzi
,
Kelvin Xu
,
Dale Schuurmans
Ensemble Adversarial Training: Attacks and Defenses
Florian Tram癡r,
Alexey Kurakin
,
Nicolas Papernot
,
Ian Goodfellow
, Dan Boneh, Patrick McDaniel
Stochastic Variational Video Prediction
Mohammad Babaeizadeh, Chelsea Finn,
Dumitru
Erhan, Roy Campbell,
Sergey Levine
Depthwise Separable Convolutions for Neural Machine Translation
Lukasz Kaiser
, Aidan N. Gomez,
Francois Chollet
Don’t Decay the Learning Rate, Increase the Batch Size
Samuel L. Smith
,
Pieter-Jan Kindermans
,
Chris Ying
,
Quoc V. Le
Generative Models of Visually Grounded Imagination
Ramakrishna Vedantam,
Ian Fischer
,
Jonathan Huang
,
Kevin Murphy
Large Scale Distributed Neural Network Training through Online Distillation
Rohan Anil
, Gabriel Pereyra,
Alexandre Passos
,
Robert Ormandi
,
George E. Dahl
,
Geoffrey E. Hinton
Learning a Neural Response Metric for Retinal Prosthesis
Nishal P. Shah
, Sasidhar Madugula, Alan Litke, Alexander Sher, EJ Chichilnisky,
Yoram Singer,
Jonathon Shlens
Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks
Shankar Krishnan
,
Ying Xiao
,
Rif A. Saurous
A Neural Representation of Sketch Drawings
David Ha
,
Douglas Eck
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling
Carlos Riquelme
,
George Tucker
,
Jasper Snoek
Generalizing Hamiltonian Monte Carlo with Neural Networks
Daniel Levy,
Matthew D. Hoffman
,
Jascha Sohl-Dickstein
Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis
Rudy Bunel, Matthew Hausknecht,
Jacob Devlin
, Rishabh Singh, Pushmeet Kohli
On the Discrimination-Generalization Tradeoff in GANs
Pengchuan Zhang, Qiang Liu,
Dengyong Zhou
, Tao Xu, Xiaodong He
A Bayesian Perspective on Generalization and Stochastic Gradient Descent
Samuel L. Smith
,
Quoc V. Le
Learning how to Explain Neural Networks: PatternNet and PatternAttribution
Pieter-Jan Kindermans
, Kristof T. Sch羹tt, Maximilian Alber, Klaus-Robert M羹ller,
Dumitru Erhan
,
Been Kim
, Sven D瓣hne
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
V穩ctor Campos,
Brendan Jou
, Xavier Gir籀-i-Nieto, Jordi Torres, Shih-Fu Chang
Towards Neural Phrase-based Machine Translation
Po-Sen Huang,
Chong Wang
, Sitao Huang,
Dengyong Zhou
, Li Deng
Unsupervised Cipher Cracking Using Discrete GANs
Aidan N. Gomez, Sicong Huang, Ivan Zhang, Bryan M. Li, Muhammad Osama,
Lukasz Kaiser
Variational Image Compression With A Scale Hyperprior
Johannes Ball矇
,
David Minnen
,
Saurabh Singh
,
Sung Jin Hwang
,
Nick Johnston
Workshop Posters
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values
Julius Adebayo
,
Justin Gilmer
,
Ian Goodfellow
,
Been Kim
Stoachastic Gradient Langevin Dynamics that Exploit Neural Network Structure
Zachary Nado
,
Jasper Snoek
, Bowen Xu, Roger Grosse, David Duvenaud, James Martens
Towards Mixed-initiative generation of multi-channel sequential structure
Anna Huang
,
Sherol Chen
, Mark J. Nelson,
Douglas Eck
Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games?
Maithra Raghu
,
Alex Irpan
, Jacob Andreas, Robert Kleinberg,
Quoc V. Le
, Jon Kleinberg
GILBO: One Metric to Measure Them All
Alexander Alemi
,
Ian Fischer
HoME: a Household Multimodal Environment
Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat,
Hugo Larochelle,
Aaron Courville
Learning to Learn without Labels
Luke Metz
,
Niru Maheswaranathan
, Brian Cheung,
Jascha Sohl-Dickstein
Learning via Social Awareness: Improving Sketch Representations with Facial Feedback
Natasha Jaques
,
Jesse Engel
,
David Ha
,
Fred Bertsch
, Rosalind Picard,
Douglas Eck
Negative Eigenvalues of the Hessian in Deep Neural Networks
Guillaume Alain,
Nicolas Le Roux
,
Pierre-Antoine Manzagol
Realistic Evaluation of Semi-Supervised Learning Algorithms
Avital Oliver
,
Augustus Odena
,
Colin Raffel
,
Ekin Cubuk
,
lan Goodfellow
Winner's Curse? On Pace, Progress, and Empirical Rigor
D. Sculley
,
Jasper Snoek
,
Alex Wiltschko
,
Ali Rahimi
Meta-Learning for Batch Mode Active Learning
Sachin Ravi,
Hugo Larochelle
To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression
Michael Zhu,
Suyog Gupta
Adversarial Spheres
Justin Gilmer
,
Luke Metz
,
Fartash Faghri
,
Sam Schoenholz
,
Maithra Raghu,
,
Martin Wattenberg
,
Ian Goodfellow
Clustering Meets Implicit Generative Models
Francesco Locatello,
Damien Vincent
, Ilya Tolstikhin, Gunnar Ratsch,
Sylvain Gelly,
Bernhard Scholkopf
Decoding Decoders: Finding Optimal Representation Spaces for Unsupervised Similarity Tasks
Vitalii Zhelezniak, Dan Busbridge, April Shen,
Samuel L. Smith
, Nils Y. Hammerla
Learning Longer-term Dependencies in RNNs with Auxiliary Losses
Trieu Trinh
,
Quoc Le
,
Andrew Dai
,
Thang Luong
Graph Partition Neural Networks for Semi-Supervised Classification
Alexander Gaunt,
Danny Tarlow
, Marc Brockschmidt, Raquel Urtasun, Renjie Liao, Richard Zemel
Searching for Activation Functions
Prajit Ramachandran
,
Barret Zoph
,
Quoc Le
Time-Dependent Representation for Neural Event Sequence Prediction
Yang Li, Nan Du
,
Samy Bengio
Faster Discovery of Neural Architectures by Searching for Paths in a Large Model
Hieu Pham, Melody Guan,
Barret Zoph
,
Quoc V. Le
,
Jeff Dean
Intriguing Properties of Adversarial Examples
Ekin Dogus Cubuk
,
Barret Zoph
,
Sam Schoenholz
,
Quoc Le
PPP-Net: Platform-aware Progressive Search for Pareto-optimal Neural Architectures
Jin-Dong Dong, An-Chieh Cheng,
Da-Cheng Juan
,
Wei Wei
, Min Sun
The Mirage of Action-Dependent Baselines in Reinforcement Learning
George Tucker
,
Surya Bhupatiraju
,
Shixiang Gu
, Richard E. Turner, Zoubin Ghahramani,
Sergey Levine
Learning to Organize Knowledge with N-Gram Machines
Fan Yang,
Jiazhong Nie
, William W. Cohen, Ni Lao
Online variance-reducing optimization
Nicolas Le Roux
, Reza Babanezhad,
Pierre-Antoine Manzagol
Expressive Speech Synthesis with Tacotron
Tuesday, March 27, 2018
Posted by Yuxuan Wang, Research Scientist and RJ Skerry-Ryan, Software Engineer, on behalf of the Machine Perception, Google Brain and TTS Research teams
At Google, we're excited about the recent rapid progress of neural network-based text-to-speech (TTS) research. In particular, end-to-end architectures, such as the
Tacotron
systems we announced last year, can both simplify voice building pipelines and produce natural-sounding speech. This will help us build better human-computer interfaces, like conversational assistants, audiobook narration, news readers, or voice design software. To deliver a truly human-like voice, however, a TTS system must learn to model
prosody
, the collection of expressive factors of speech, such as intonation, stress, and rhythm. Most current end-to-end systems, including Tacotron, don't explicitly model prosody, meaning they can't control exactly how the generated speech should sound. This may lead to monotonous-sounding speech, even when models are trained on very expressive datasets like audiobooks, which often contain character voices with significant variation. Today, we are excited to share two new papers that address these problems.
Our first paper, “
Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
”, introduces the concept of a
prosody embedding
. We augment the Tacotron architecture with an additional prosody encoder that computes a low-dimensional embedding from a clip of human speech (the reference audio).
We augment Tacotron with a prosody encoder. The lower half of the image is the original Tacotron sequence-to-sequence model. For technical details, please refer to
the paper
.
This embedding captures characteristics of the audio that are independent of phonetic information and idiosyncratic speaker traits — these are attributes like stress, intonation, and timing. At inference time, we can use this embedding to perform prosody transfer, generating speech in the voice of a completely different speaker, but exhibiting the prosody of the reference.
Text:
*Is*
that Utah travel agency?
Reference prosody (Australian)
Synthesized without prosody embedding (American)
Synthesized with prosody embedding (American)
The embedding can also transfer fine time-aligned prosody from one phrase to a slightly different phrase, though this technique works best when the reference and target phrases are similar in length and structure.
Reference Text: For the first time in her life she had been danced tired.
Synthesized Text: For the last time in his life he had been handily embarrassed.
Reference prosody (American)
Synthesized without prosody embedding (American)
Synthesized with prosody embedding (American)
Excitingly, we observe prosody transfer even when the reference audio comes from a speaker whose voice is not in Tacotron's training data.
Text: I've Swallowed a Pollywog.
Reference prosody (Unseen American Speaker)
Synthesized without prosody embedding (British)
Synthesized with prosody embedding (British)
This is a promising result, as it paves the way for voice interaction designers to use their own voice to customize speech synthesis. You can listen to the full set of audio demos for “
Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
” on
this web page
.
Despite their ability to transfer prosody with high fidelity, the embeddings from the paper above don't completely disentangle prosody from the content of a reference audio clip. (This explains why they transfer prosody best to phrases of similar structure and length.) Furthermore, they require a clip of reference audio at inference time. A natural question then arises: can we develop a model of expressive speech that alleviates these problems?
In our second paper, “
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
”, we do just that. Building upon the architecture in our first paper, we propose a new unsupervised method for modeling latent "factors" of speech. The key to this model is that, rather than learning fine time-aligned prosodic elements, it learns higher-level speaking style patterns that can be transferred across arbitrarily different phrases.
The model works by adding an extra attention mechanism to Tacotron, forcing it to represent the prosody embedding of any speech clip as the linear combination of a fixed set of basis embeddings. We call these embeddings
Global Style Tokens
(GSTs), and find that they learn text-independent variations in a speaker's style (soft, high-pitch, intense, etc.), without the need for explicit style labels.
Model architecture of Global Style Tokens. The prosody embedding is decomposed into “style tokens” to enable unsupervised style control and transfer. For technical details, please refer to
the paper
.
At inference time, we can select or modify the combination weights for the tokens, allowing us to force Tacotron to use a specific speaking style without needing a reference audio clip. Using GSTs, for example, we can make different sentences of varying lengths sound more "lively", "angry", "lamenting", etc:
Text: United Airlines five six three from Los Angeles to New Orleans has Landed.
Style 1
Style 2
Style 3
Style 4
Style 5
The text-independent nature of GSTs make them ideal for
style transfer
, which takes a reference audio clip spoken in a specific style and transfers its style to any target phrase we choose. To achieve this, we first run inference to predict the GST combination weights for an utterance whose style we want to imitate. We can then feed those combination weights to the model to synthesize completely different phrases — even those with very different lengths and structure — in the same style.
Finally, our paper shows that Global Style Tokens can model more than just speaking style. When trained on noisy YouTube audio from unlabeled speakers, a GST-enabled Tacotron learns to represent noise sources and distinct speakers as separate tokens. This means that by selecting the GSTs we use in inference, we can synthesize speech free of background noise, or speech in the voice of a specific unlabeled speaker from the dataset. This exciting result provides a path towards highly scalable but robust speech synthesis. You can listen to the full set of demos for "
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
" on
this web page
.
We are excited about the potential applications and opportunities that these two bodies of research enable. In the meantime, there are new important research problems to be addressed. We'd like to extend the techniques of the first paper to support prosody transfer in the natural pitch range of the target speaker. We'd also like to develop techniques to select appropriate prosody or speaking style automatically from context, using, for example, the integration of natural language understanding with TTS. Finally, while our first paper proposes an initial set of objective and subjective metrics for prosody transfer, we'd like to develop these further to help establish generally-accepted methods for prosodic evaluation.
Acknowledgements
These projects were done jointly between multiple Google teams. Contributors include RJ Skerry-Ryan, Yuxuan Wang, Daisy Stanton, Eric Battenberg, Ying Xiao, Joel Shor, Rif A. Saurous, Yu Zhang, Ron J. Weiss, Rob Clark, Fei Ren and Ye Jia.
Using Evolutionary AutoML to Discover Neural Network Architectures
Thursday, March 15, 2018
Posted by Esteban Real, Senior Software Engineer, Google Brain Team
The brain has evolved over a long time, from very simple worm brains
500 million years ago
to a diversity of modern structures today. The human brain, for example, can accomplish a wide variety of activities, many of them effortlessly — telling whether a visual scene contains animals or buildings feels trivial to us, for example. To perform activities like these,
artificial neural networks
require careful design by experts over years of difficult research, and typically address one specific task, such as to
find what's in a photograph
, to
call a genetic variant
, or to
help diagnose a disease
. Ideally, one would want to have an automated method to generate the right architecture for any given task.
One approach to generate these architectures is through the use of evolutionary algorithms. Traditional research into neuro-evolution of topologies (e.g.
Stanley and Miikkulainen 2002
) has laid the foundations that allow us to apply these algorithms at scale today, and many groups are working on the subject, including
OpenAI
,
Uber Labs
,
Sentient Labs
and
DeepMind
. Of course, the
Google Brain team
has been thinking about
AutoML
too. In addition to learning-based approaches (eg.
reinforcement learning
), we wondered if we could use our computational resources to programmatically
evolve
image classifiers at unprecedented scale. Can we achieve solutions with minimal expert participation? How good can today's artificially-evolved neural networks be? We address these questions through two papers.
In “
Large-Scale Evolution of Image Classifiers
,” presented at
ICML 2017
, we set up an evolutionary process with simple building blocks and trivial initial conditions. The idea was to "sit back" and let evolution at scale do the work of constructing the architecture. Starting from very simple networks, the process found classifiers comparable to hand-designed models at the time. This was encouraging because many applications may require little user participation. For example, some users may need a better model but may not have the time to become machine learning experts. A natural question to consider next was whether a combination of hand-design and evolution could do better than either approach alone. Thus, in our more recent paper, “
Regularized Evolution for Image Classifier Architecture Search
” (2018), we participated in the process by providing sophisticated building blocks and good initial conditions (discussed below). Moreover, we scaled up computation using Google's new
TPUv2 chips
. This combination of modern hardware, expert knowledge, and evolution worked together to produce state-of-the-art models on
CIFAR-10
and
ImageNet
, two popular benchmarks for image classification.
A Simple Approach
The following is an example of an experiment from our first paper. In the figure below, each dot is a neural network trained on the
CIFAR-10
dataset, which is commonly used to train image classifiers. Initially, the population consists of one thousand identical simple
seed
models (no hidden layers). Starting from simple seed models is important — if we had started from a high-quality model with initial conditions containing expert knowledge, it would have been easier to get a high-quality model in the end. Once seeded with the simple models, the process advances in steps. At each step, a pair of neural networks is chosen at random. The network with higher accuracy is selected as a
parent
and is copied and mutated to generate a
child
that is then added to the population, while the other neural network
dies
out. All other networks remain unchanged during the step. With the application of many such steps in succession, the population evolves.
Progress of an evolution experiment. Each dot represents an individual in the population. The four diagrams show examples of discovered architectures. These correspond to the best individual (rightmost; selected by validation accuracy) and three of its ancestors.
The mutations in our first paper are purposefully simple: remove a convolution at random, add a skip connection between arbitrary layers, or change the learning rate, to name a few. This way, the results show the potential of the evolutionary algorithm, as opposed to the quality of the search space. For example, if we had used a single mutation that transforms one of the seed networks into an
Inception-ResNet
classifier in one step, we would be incorrectly concluding that the algorithm found a good answer. Yet, in that case, all we would have done is hard-coded the final answer into a complex mutation, rigging the outcome. If instead we stick with simple mutations, this cannot happen and evolution is truly doing the job. In the experiment in the figure,
simple
mutations and the selection process cause the networks to improve over time and reach high test accuracies, even though the test set had never been seen during the process. In this paper, the networks can also inherit their parent's weights. Thus, in addition to evolving the architecture, the population trains its networks while exploring the search space of initial conditions and learning-rate schedules. As a result, the process yields fully trained models with optimized hyperparameters. No expert input is needed after the experiment starts.
In all the above, even though we were minimizing the researcher's participation by having simple initial architectures and intuitive mutations, a good amount of expert knowledge went into the building blocks those architectures were made of. These included important inventions such as convolutions, ReLUs and batch-normalization layers. We were evolving an
architecture
made up of these components. The term "architecture" is not accidental: this is analogous to constructing a house with high-quality bricks.
Combining Evolution and Hand Design
After our first paper, we wanted to reduce the search space to something more manageable by giving the algorithm fewer choices to explore. Using our architectural analogy, we removed all the possible ways of making large-scale errors, such as putting the wall above the roof, from the search space. Similarly with neural network architecture searches, by fixing the large-scale structure of the network, we can help the algorithm out. So how to do this? The inception-like modules introduced in
Zoph et al.
(2017) for the purpose of architecture search proved very powerful. Their idea is to have a deep
stack
of repeated modules called
cells
. The stack is fixed but the architecture of the individual modules can change.
The building blocks introduced in
Zoph et al.
(2017). The diagram on the left is the outer structure of the full neural network, which parses the input data from bottom to top through a stack of repeated cells. The diagram on the right is the inside structure of a cell. The goal is to find a cell that yields an accurate network.
In our second paper, “
Regularized Evolution for Image Classifier Architecture Search
” (2018), we presented the results of applying evolutionary algorithms to the search space described above. The mutations modify the cell by randomly reconnecting the inputs (the arrows on the right diagram in the figure) or randomly replacing the operations (for example, they can replace the "max 3x3" in the figure, a max-pool operation, with an arbitrary alternative). These mutations are still relatively simple, but the initial conditions are not: the population is now initialized with models that must conform to the outer stack of cells, which was designed by an expert. Even though the cells in these seed models are random, we are no longer starting from simple models, which makes it easier to get to high-quality models in the end. If the evolutionary algorithm is contributing meaningfully, the final networks should be significantly better than the networks we already know can be constructed within this search space. Our paper shows that evolution can indeed find state-of-the-art models that either match or outperform hand-designs.
A Controlled Comparison
Even though the mutation/selection evolutionary process is not complicated, maybe an even more straightforward approach (like random search) could have done the same. Other alternatives, though not simpler, also exist in the literature (like reinforcement learning). Because of this, the main purpose of our second paper was to provide a controlled comparison between techniques.
Comparison between evolution, reinforcement learning, and random search for the purposes of architecture search. These experiments were done on the CIFAR-10 dataset, under the same conditions as
Zoph et al.
(2017), where the search space was originally used with reinforcement learning.
The figure above compares evolution, reinforcement learning, and random search. On the left, each curve represents the progress of an experiment, showing that evolution is faster than reinforcement learning in the earlier stages of the search. This is significant because with less compute power available, the experiments may have to stop early. Moreover evolution is quite robust to changes in the dataset or search space. Overall, the goal of this controlled comparison is to provide the research community with the results of a computationally expensive experiment. In doing so, it is our hope to facilitate architecture searches for everyone by providing a case study of the relationship between the different search algorithms. Note, for example, that the figure above shows that the final models obtained with evolution can reach very high accuracy while using fewer floating-point operations.
One important feature of the evolutionary algorithm we used in our second paper is a form of
regularization
: instead of letting the worst neural networks die, we remove the oldest ones — regardless of how good they are. This improves robustness to changes in the task being optimized and tends to produce more accurate networks in the end. One reason for this may be that since we didn't allow weight inheritance, all networks must train from scratch. Therefore, this form of regularization selects for networks that remain good when they are re-trained. In other words, because a model can be more accurate just by chance — noise in the training process means even identical architectures may get different accuracy values — only architectures that remain accurate through the generations will survive in the long run, leading to the selection of networks that retrain well. More details of this conjecture can be found in
the paper
.
The state-of-the-art models we evolved are nicknamed
AmoebaNets
, and are one of the latest results from our AutoML efforts. All these experiments took a lot of computation — we used hundreds of GPUs/TPUs for days. Much like a single modern computer can outperform thousands of decades-old machines, we hope that in the future these experiments will become household. Here we aimed to provide a glimpse into that future.
Acknowledgements
We would like to thank Alok Aggarwal, Yanping Huang, Andrew Selle, Sherry Moore, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Alex Kurakin, Quoc Le, Barret Zoph, Jon Shlens, Vijay Vasudevan, Vincent Vanhoucke, Megan Kacholia, Jeff Dean, and the rest of the
Google Brain team
for the collaborations that made this work possible.
Tacotron 2: Generating Human-like Speech from Text
Tuesday, December 19, 2017
Posted by Jonathan Shen and Ruoming Pang, Software Engineers, on behalf of the Google Brain and Machine Perception Teams
Generating very natural sounding speech from text (text-to-speech, TTS) has been a research goal for decades. There has been great progress in TTS research over the last few years and many individual pieces of a complete TTS system have greatly improved. Incorporating ideas from past work such as
Tacotron
and
WaveNet
, we added more improvements to end up with our new system,
Tacotron 2
. Our approach does not use complex linguistic and acoustic features as input. Instead, we generate human-like speech from text using neural networks trained using only speech examples and corresponding text transcripts.
A full description of our new system can be found in our paper “
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
.” In a nutshell it works like this: We use a sequence-to-sequence model optimized for TTS to map a sequence of letters to a sequence of features that encode the audio. These features, an 80-dimensional audio spectrogram with frames computed every 12.5 milliseconds, capture not only pronunciation of words, but also various subtleties of human speech, including volume, speed and intonation. Finally these features are converted to a 24 kHz waveform using a
WaveNet
-like architecture.
A detailed look at Tacotron 2's model architecture. The lower half of the image describes the sequence-to-sequence model that maps a sequence of letters to a spectrogram. For technical details, please refer to
the paper
.
You can listen to some of the
Tacotron 2 audio samples
that demonstrate the results of our state-of-the-art TTS system. In an evaluation where we asked human listeners to rate the naturalness of the generated speech, we obtained a score that was comparable to that of professional recordings.
While our samples sound great, there are still some difficult problems to be tackled. For example, our system has difficulties pronouncing complex words (such as “
decorum
” and “
merlot
”), and in extreme cases it can even randomly generate strange noises. Also, our system cannot yet generate audio in realtime. Furthermore, we cannot yet control the generated speech, such as directing it to sound happy or sad. Each of these is an interesting research problem on its own.
Acknowledgements
Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, Yonghui Wu, Sound Understanding team, TTS Research team, and TensorFlow team.
Google at NIPS 2017
Sunday, December 03, 2017
Posted by Christian Howard, Editor-in-Chief, Research Communications
This week, Long Beach, California hosts the 31
st
annual
Conference on Neural Information Processing Systems
(NIPS 2017), a machine learning and computational neuroscience conference that includes invited talks, demonstrations and presentations of some of the latest in machine learning research. Google will have a strong presence at NIPS 2017, with over 450 Googlers attending to contribute to, and learn from, the broader academic research community via technical talks and posters, workshops, competitions and tutorials.
Google is at the forefront of machine learning, actively exploring virtually all aspects of the field from classical algorithms to deep learning and more. Focusing on both theory and application, much of our work on language understanding, speech, translation, visual processing, and prediction relies on state-of-the-art techniques that push the boundaries of what is possible. In all of those tasks and many others, we develop learning approaches to understand and generalize, providing us with new ways of looking at old problems and helping transform how we work and live.
If you are attending NIPS 2017, we hope you’ll stop by our booth and chat with our researchers about the projects and opportunities at Google that go into solving interesting problems for billions of people, and to see demonstrations of some of the exciting research we pursue. You can also learn more about our work being presented in the list below (Googlers highlighted in
blue
).
Google is a Platinum Sponsor of NIPS 2017.
Organizing Committee
Program Chair:
Samy Bengio
Senior Area Chairs include:
Corinna Cortes
,
Dale Schuurmans
,
Hugo Larochelle
Area Chairs include:
Afshin Rostamizadeh
,
Amir Globerson
,
Been Kim
,
D. Sculley
,
Dumitru Erhan
,
Gal Chechik
,
Hartmut Neven
,
Honglak Lee
,
Ian Goodfellow
,
Jasper Snoek
,
John Wright
,
Jon Shlens
,
Lihong Li
,
Maya Gupta
,
Moritz Hardt
,
Navdeep Jaitly
,
Ryan Adams
,
Sally Goldman
,
Sanjiv Kumar
,
Surya Ganguli
,
Tara Sainath
,
Umar Syed
,
Viren Jain
,
Vitaly Kuznetsov
Invited Talk
Powering the next 100 years
John Platt
Accepted Papers
A Meta-Learning Perspective on Cold-Start Recommendations for Items
Manasi Vartak,
Hugo Larochelle
, Arvind Thiagarajan
AdaGAN: Boosting Generative Models
Ilya Tolstikhin,
Sylvain Gelly
,
Olivier Bousquet
, Carl-Johann Simon-Gabriel, Bernhard Sch繹lkopf
Deep Lattice Networks and Partial Monotonic Functions
Seungil You
,
David Ding
,
Kevin Canini
,
Jan Pfeifer
,
Maya Gupta
From which world is your graph
Cheng Li, Varun Kanade,
Felix MF Wong
, Zhenming Liu
Hiding Images in Plain Sight: Deep Steganography
Shumeet Baluja
Improved Graph Laplacian via Geometric Self-Consistency
Dominique Joncas
, Marina Meila, James McQueen
Model-Powered Conditional Independence Test
Rajat Sen,
Ananda Theertha Suresh
, Karthikeyan Shanmugam, Alexandros Dimakis, Sanjay Shakkottai
Nonlinear random matrix theory for deep learning
Jeffrey Pennington
,
Pratik Worah
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Jeffrey Pennington
,
Samuel Schoenholz
, Surya Ganguli
SGD Learns the Conjugate Kernel Class of the Network
Amit Daniely
SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
Maithra Raghu
,
Justin Gilmer
, Jason Yosinski,
Jascha Sohl-Dickstein
Learning Hierarchical Information Flow with Recurrent Neural Modules
Danijar Hafner,
Alexander Irpan
,
James Davidson
, Nicolas Heess
Online Learning with Transductive Regret
Scott Yang,
Mehryar Mohri
Acceleration and Averaging in Stochastic Descent Dynamics
Walid Krichene
, Peter Bartlett
Parameter-Free Online Learning via Model Selection
Dylan J Foster,
Satyen Kale
,
Mehryar Mohri
, Karthik Sridharan
Dynamic Routing Between Capsules
Sara Sabour
,
Nicholas Frosst
,
Geoffrey E Hinton
Modulating early visual processing by language
Harm de Vries, Florian Strub, Jeremie Mary,
Hugo Larochelle
, Olivier Pietquin, Aaron C Courville
MarrNet: 3D Shape Reconstruction via 2.5D Sketches
Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun,
Bill Freeman
, Josh Tenenbaum
Affinity Clustering: Hierarchical Clustering at Scale
Mahsa Derakhshan, Soheil Behnezhad,
Mohammadhossein Bateni
,
Vahab Mirrokni
, MohammadTaghi Hajiaghayi,
Silvio Lattanzi
,
Raimondas Kiveris
Asynchronous Parallel Coordinate Minimization for MAP Inference
Ofer Meshi
, Alexander Schwing
Cold-Start Reinforcement Learning with Softmax Policy Gradient
Nan Ding
,
Radu Soricut
Filtering Variational Objectives
Chris J Maddison,
Dieterich Lawson
,
George Tucker
,
Mohammad Norouzi
, Nicolas Heess, Andriy Mnih, Yee Whye Teh, Arnaud Doucet
Multi-Armed Bandits with Metric Movement Costs
Tomer Koren
, Roi Livni,
Yishay Mansour
Multiscale Quantization for Fast Similarity Search
Xiang Wu
,
Ruiqi Guo
,
Ananda Theertha Suresh
,
Sanjiv Kumar
,
Daniel Holtmann-Rice
,
David Simcha
,
Felix Yu
Reducing Reparameterization Gradient Variance
Andrew Miller, Nicholas Foti, Alexander D'Amour,
Ryan Adams
Statistical Cost Sharing
Eric Balkanski,
Umar Syed
,
Sergei Vassilvitskii
The Unreasonable Effectiveness of Structured Random Orthogonal Embeddings
Krzysztof Choromanski
, Mark Rowland, Adrian Weller
Value Prediction Network
Junhyuk Oh, Satinder Singh,
Honglak Lee
REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
George Tucker
, Andriy Mnih, Chris J Maddison,
Dieterich Lawson
,
Jascha Sohl-Dickstein
Approximation and Convergence Properties of Generative Adversarial Learning
Shuang Liu,
Olivier Bousquet
, Kamalika Chaudhuri
Attention is All you Need
Ashish Vaswani
,
Noam Shazeer
,
Niki Parmar
,
Jakob Uszkoreit
,
Llion Jones
, Aidan N Gomez,
ukasz Kaiser
,
Illia Polosukhin
PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference
Jonathan Huggins,
Ryan Adams
, Tamara Broderick
Repeated Inverse Reinforcement Learning
Kareem Amin
, Nan Jiang, Satinder Singh
Fair Clustering Through Fairlets
Flavio Chierichetti,
Ravi Kumar
,
Silvio Lattanzi
,
Sergei Vassilvitskii
Affine-Invariant Online Optimization and the Low-rank Experts Problem
Tomer Koren
, Roi Livni
Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
Sergey Ioffe
Bridging the Gap Between Value and Policy Based Reinforcement Learning
Ofir Nachum
, Mohammad Norouzi,
Kelvin Xu
,
Dale Schuurmans
Discriminative State Space Models
Vitaly Kuznetsov
,
Mehryar Mohri
Dynamic Revenue Sharing
Santiago Balseiro,
Max Lin
,
Vahab Mirrokni
,
Renato Leme
, Song Zuo
Multi-view Matrix Factorization for Linear Dynamical System Estimation
Mahdi Karami, Martha White,
Dale Schuurmans
, Csaba Szepesvari
On Blackbox Backpropagation and Jacobian Sensing
Krzysztof Choromanski
,
Vikas Sindhwani
On the Consistency of Quick Shift
Heinrich Jiang
Revenue Optimization with Approximate Bid Predictions
Andres Munoz
,
Sergei Vassilvitskii
Shape and Material from Sound
Zhoutong Zhang, Qiujia Li, Zhengjia Huang, Jiajun Wu, Josh Tenenbaum,
Bill Freeman
Learning to See Physics via Visual De-animation
Jiajun Wu, Erika Lu, Pushmeet Kohli,
Bill Freeman
, Josh Tenenbaum
Conference Demos
Electronic Screen Protector with Efficient and Robust Mobile Vision
Hee Jung Ryu
,
Florian Schroff
Magenta and deeplearn.js: Real-time Control of DeepGenerative Music Models in the Browser
Curtis Hawthorne
,
Ian Simon
,
Adam Roberts
,
Jesse Engel
,
Daniel Smilkov
,
Nikhil Thorat
,
Douglas Eck
Workshops
6th Workshop on Automated Knowledge Base Construction
(AKBC) 2017
Program Committee includes:
Arvind Neelakanta
Authors include:
Jiazhong Nie
,
Ni Lao
Acting and Interacting in the Real World: Challenges in Robot Learning
Invited Speakers include:
Pierre Sermanet
Advances in Approximate Bayesian Inference
Panel moderator:
Matthew D. Hoffman
Conversational AI - Today's Practice and Tomorrow's Potential
Invited Speakers include:
Matthew Henderson
,
Dilek Hakkani-Tur
Organizers include:
Larry Heck
Extreme Classification: Multi-class and Multi-label Learning in Extremely Large Label Spaces
Invited Speakers include:
Ed Chi
,
Mehryar Mohri
Learning in the Presence of Strategic Behavior
Invited Speakers include:
Mehryar Mohri
Presenters include:
Andres Munoz Medina
,
Sebastien Lahaie
,
Sergei Vassilvitskii
,
Balasubramanian Sivan
Learning on Distributions, Functions, Graphs and Groups
Invited speakers include:
Corinna Cortes
Machine Deception
Organizers include:
Ian Goodfellow
Invited Speakers include:
Jacob Buckman
,
Aurko Roy
,
Colin Raffel
,
Ian Goodfellow
Machine Learning and Computer Security
Invited Speakers include:
Ian Goodfellow
Organizers include:
Nicolas Papernot
Authors include:
Jacob Buckman
,
Aurko Roy
,
Colin Raffel
,
Ian Goodfellow
Machine Learning for Creativity and Design
Keynote Speakers include:
Ian Goodfellow
Organizers include:
Doug Eck
,
David Ha
Machine Learning for Audio Signal Processing (ML4Audio)
Authors include:
Aren Jansen
,
Manoj Plakal
,
Dan Ellis
,
Shawn Hershey
,
Channing Moore
,
Rif A. Saurous
,
Yuxuan Wang
,
RJ Skerry-Ryan
,
Ying Xiao
,
Daisy Stanton
,
Joel Shor
,
Eric Batternberg
,
Rob Clark
Machine Learning for Health (ML4H)
Organizers include:
Jasper Snoek
,
Alex Wiltschko
Keynote:
Fei-Fei Li
NIPS Time Series Workshop 2017
Organizers include:
Vitaly Kuznetsov
Authors include:
Brendan Jou
OPT 2017: Optimization for Machine Learning
Organizers include:
Sashank Reddi
ML Systems Workshop
Invited Speakers include:
Rajat Monga
,
Alexander Mordvintsev
,
Chris Olah
,
Jeff Dean
Authors include:
Alex Beutel
,
Tim Kraska
,
Ed H. Chi
,
D. Scully
,
Michael Terry
Aligned Artificial Intelligence
Invited Speakers include:
Ian Goodfellow
Bayesian Deep Learning
Organizers include:
Kevin Murphy
Invited speakers include:
Nal Kalchbrenner
,
Matthew D. Hoffman
BigNeuro 2017
Invited speakers include:
Viren Jain
Cognitively Informed Artificial Intelligence: Insights From Natural Intelligence
Authors include:
Jiazhong Nie
,
Ni Lao
Deep Learning At Supercomputer Scale
Organizers include:
Erich Elsen
,
Zak Stone
,
Brennan Saeta
,
Danijar Haffner
Deep Learning: Bridging Theory and Practice
Invited Speakers include:
Ian Goodfellow
Interpreting, Explaining and Visualizing Deep Learning
Invited Speakers include:
Been Kim
,
Honglak Lee
Authors include:
Pieter Kinderman
,
Sara Hooker
,
Dumitru Erhan
,
Been Kim
Learning Disentangled Features: from Perception to Control
Organizers include:
Honglak Lee
Authors include:
Jasmine Hsu
,
Arkanath Pathak
,
Abhinav Gupta
,
James Davidson
,
Honglak Lee
Learning with Limited Labeled Data: Weak Supervision and Beyond
Invited Speakers include:
Ian Goodfellow
Machine Learning on the Phone and other Consumer Devices
Invited Speakers include:
Rajat Monga
Organizers include:
Hrishikesh Aradhye
Authors include:
Suyog Gupta
,
Sujith Ravi
Optimal Transport and Machine Learning
Organizers include:
Olivier Bousquet
The future of gradient-based machine learning software & techniques
Organizers include:
Alex Wiltschko
,
Bart van Merri禱nboer
Workshop on Meta-Learning
Organizers include:
Hugo Larochelle
Panelists include:
Samy Bengio
Authors include:
Aliaksei Severyn
,
Sascha Rothe
Symposiums
Deep Reinforcement Learning Symposium
Authors include:
Benjamin Eysenbach
,
Shane Gu
,
Julian Ibarz
,
Sergey Levine
Interpretable Machine Learning
Authors include:
Minmin Chen
Metalearning
Organizers include:
Quoc V Le
Competitions
Adversarial Attacks and Defences
Organizers include:
Alexey Kurakin
,
Ian Goodfellow
,
Samy Bengio
Competition IV: Classifying Clinically Actionable Genetic Mutations
Organizers include:
Wendy Kan
Tutorial
Fairness in Machine Learning
Solon Barocas
,
Moritz Hardt
Labels
accessibility
ACL
ACM
Acoustic Modeling
Adaptive Data Analysis
ads
adsense
adwords
Africa
AI
Algorithms
Android
Android Wear
API
App Engine
App Inventor
April Fools
Art
Audio
Augmented Reality
Australia
Automatic Speech Recognition
Awards
Cantonese
Chemistry
China
Chrome
Cloud Computing
Collaboration
Computational Imaging
Computational Photography
Computer Science
Computer Vision
conference
conferences
Conservation
correlate
Course Builder
crowd-sourcing
CVPR
Data Center
Data Discovery
data science
datasets
Deep Learning
DeepDream
DeepMind
distributed systems
Diversity
Earth Engine
economics
Education
Electronic Commerce and Algorithms
electronics
EMEA
EMNLP
Encryption
entities
Entity Salience
Environment
Europe
Exacycle
Expander
Faculty Institute
Faculty Summit
Flu Trends
Fusion Tables
gamification
Gboard
Gmail
Google Accelerated Science
Google Books
Google Brain
Google Cloud Platform
Google Docs
Google Drive
Google Genomics
Google Maps
Google Photos
Google Play Apps
Google Science Fair
Google Sheets
Google Translate
Google Trips
Google Voice Search
Google+
Government
grants
Graph
Graph Mining
Hardware
HCI
Health
High Dynamic Range Imaging
ICLR
ICML
ICSE
Image Annotation
Image Classification
Image Processing
Inbox
India
Information Retrieval
internationalization
Internet of Things
Interspeech
IPython
Journalism
jsm
jsm2011
K-12
KDD
Keyboard Input
Klingon
Korean
Labs
Linear Optimization
localization
Low-Light Photography
Machine Hearing
Machine Intelligence
Machine Learning
Machine Perception
Machine Translation
Magenta
MapReduce
market algorithms
Market Research
Mixed Reality
ML
MOOC
Moore's Law
Multimodal Learning
NAACL
Natural Language Processing
Natural Language Understanding
Network Management
Networks
Neural Networks
Nexus
Ngram
NIPS
NLP
On-device Learning
open source
operating systems
Optical Character Recognition
optimization
osdi
osdi10
patents
Peer Review
ph.d. fellowship
PhD Fellowship
PhotoScan
Physics
PiLab
Pixel
Policy
Professional Development
Proposals
Public Data Explorer
publication
Publications
Quantum AI
Quantum Computing
renewable energy
Research
Research Awards
resource optimization
Robotics
schema.org
Search
search ads
Security and Privacy
Semantic Models
Semi-supervised Learning
SIGCOMM
SIGMOD
Site Reliability Engineering
Social Networks
Software
Speech
Speech Recognition
statistics
Structured Data
Style Transfer
Supervised Learning
Systems
TensorBoard
TensorFlow
TPU
Translate
trends
TTS
TV
UI
University Relations
UNIX
User Experience
video
Video Analysis
Virtual Reality
Vision Research
Visiting Faculty
Visualization
VLDB
Voice Search
Wiki
wikipedia
WWW
YouTube
Archive
2018
May
Apr
Mar
Feb
Jan
2017
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2016
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2015
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2014
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2012
Dec
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2011
Dec
Nov
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2010
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2009
Dec
Nov
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2008
Dec
Nov
Oct
Sep
Jul
May
Apr
Mar
Feb
2007
Oct
Sep
Aug
Jul
Jun
Feb
2006
Dec
Nov
Sep
Aug
Jul
Jun
Apr
Mar
Feb
Feed
Google
on
Follow @googleresearch
Give us feedback in our
Product Forums
.