Publications
Academic research and publications

Research areas
- Content Understanding
- Recommender Systems
- User Modeling
- Graph Learning
- ML Ethics
- Other Publications
Content Understanding
Using machine learning to understand content to can classify, filter, and improve recommendation performance.
LMSOC: An Approach for Socially Sensitive Pretraining
EMNLP
We propose a simple but effective approach to incorporate speaker social context into the learned representations of large-scale language models. Our method first learns dense representations of social contexts using graph representation learning algorithms and then primes language model pretraining with these social context representations. We evaluate our approach on geographically-sensitive language-modeling tasks and show a substantial improvement (more than 100% relative lift on MRR) compared to baselines.
Authors:
Vivek Kulkarni, Shubhanshu Mishra, Aria Haghighi
Publication date: 10/20/2021
Improved Multilingual Language Model Pretraining for Social Media Text via Translation Pair Prediction
EMNLP
We evaluate a simple approach to improving zero-shot multilingual transfer of mBERT on social media corpus by adding a pretraining task called translation pair prediction (TPP), which predicts whether a pair of cross-lingual texts are a valid translation.
Authors:
Shubhanshu Mishra, Aria Haghighi
Publication date: 10/20/2021
Putting words into the system’s mouth: A targeted attack on neural
ACL
Neural machine translation systems are known to be vulnerable to adversarial test inputs, however, as we show in this paper, these systems are also vulnerable to training attacks. Specifically, we propose a poisoning attack in which a malicious adversary inserts a small poisoned sample of monolingual text into the training set of a system trained using back-translation.
Authors:
Jun Wang, Chang Xu, Francisco Guzman , Ahmed El-Kishky, Yuqing Tang, Benjamin I. P. Rubinstein, Trevor Cohn
Publication date: 2021
Adapting High-resource NMT Models to Translate Low-resource Related
ACL
In this work, we exploit this linguistic overlap to facilitate translating to and from a low-resource language with only monolingual data, in addition to any parallel data in the related high-resource language. Our method, NMT-Adapt, combines denoising autoencoding, back-translation and adversarial objectives to utilize monolingual data for low-resource adaptation.
Authors:
Wei-Jen Ko, Ahmed El-Kishky, Adithya Renduchintala, Vishrav Chaudhary, Naman Goyal, Francisco Guzmán, Pascale Fung, Philipp Koehn, Mona Diab
Publication date: 2021
XLEnt: Mining a Large Cross-lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignment
EMNLP
We propose Lexical-Semantic-Phonetic Align (LSP-Align), a technique to automatically mine cross-lingual entity lexica from mined web data. We demonstrate LSP-Align outperforms baselines at extracting cross-lingual entity pairs and mine 164 million entity pairs from 120 different languages aligned with English.
Authors:
Ahmed El-Kishky, Adithya Renduchintala, James Cross, Francisco Guzmán, Philipp Koehn
Publication date: 2021
Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications
EMNLP
In this work, we evaluate several model compression techniques for QE and find that, despite their popularity in other NLP tasks, they lead to poor performance in this regression setting. We observe that a full model parameterization is required to achieve SoTA results in a regression task.
Authors:
Shuo Sun, Ahmed El-Kishky, Vishrav Chaudhary, James Cross, Francisco Guzmán, Lucia Specia
Publication date: 2021
Discriminative Topic Modeling with Logistic LDA
NeurlIPS
We propose logistic LDA, a novel discriminative variant of latent Dirichlet allocation which is easy to apply to arbitrary inputs.
Authors:
Iryna Korshunova, Hanchen Xiong, Mateusz Fedoryszak, Lucas Theis
Publication date: 2019
Smile, be Happy :) Emoji Embedding for Visual Sentiment Analysis
ICCV (CroMol workshop)
In this work, we propose to overcome this problem by learning a novel sentiment-aligned image embedding that is better suited for subsequent visual sentiment analysis.
Authors:
Ziad Al-Halah, Andrew Aitken, Wenzhe Shi, Jose Caballero
Publication date: 2019
CARL: Aggregated Search with Context-Aware Module Embedding Learning
IJCNN
Our model applies a recurrent neural network with attention mechanism to encode the context, and incorporates the encoded context information into module embeddings. The context-aware module embeddings together with the ranking policy are jointly optimized under the Markov decision process (MDP) formulation.
Authors:
Xiaojie Wang, Jianzhong Qi, Yu Sun, Rui Zhang, Hai-Tao Zheng4
Publication date: 2019
Learning what and where to attend
ICLR
Here, we demonstrate the benefit of using stronger supervisory signals by teaching DCNs to attend to image regions that humans deem important for object recognition.
Authors:
Drew Linsley, Dan Shiebler, Sven Eberhardt, Thomas Serre
Publication date: 2019
Fighting Redundancy and Model Decay with Embeddings
SigKDD CMI
In this paper, we detail the commoditized tools, algorithms and pipelines that we have developed and are developing at Twitter to regularly generate high quality, up-to-date embeddings and share them broadly across the company.
Authors:
Dan Shiebler, Luca Belli, Jay Baxter, Hanchen Xiong, Abhishek Tayal
Publication date: 2018
Adaptive Paired-Comparison Method for Subjective Video Quality Assessment on Mobile Devices
PCS
To effectively evaluate subjective visual quality in weakly-controlled environments, we propose an Adaptive Paired Comparison method based on particle filtering. As our approach requires each sample to be rated only once, the test time compared to regular paired comparison can be reduced.
Authors:
Katherine Storrs, Sebastiaan Van Leuven, Steve Kojder, Lucas Theis, Ferenc Huszár
Publication date: 2018
Learning Effective Embeddings for Machine Generated Emails with Applications to Email Category Prediction
IEEE International Conference on Big Data (BigData)
We propose a general framework for learning embeddings for emails and users, using as input only the sequence of B2C templates users receive and open. (A template is a B2C email stripped of all transient information related to specific users.
Authors:
Yu Sun, Lluis Garcia-Pueyo, James B. Wendt, Marc Najork, Andrei Broder
Publication date: 2018
When Hashes Met Wedges: A Distributed Algorithm for Finding High Similarity Vectors
WWW
Our work directly addresses this challenge by introducing a new algorithm — WHIMP — that solves this problem efficiently in the MapReduce model. The key insight in WHIMP is to combine the “wedge-sampling” approach of Cohen-Lewis for approximate matrix multiplication with the SimHash random projection techniques of Charikar.
Authors:
Aneesh Sharma, C. Seshadhri, Ashish Goel
Publication date: 2017
Recommender Systems
Improving the performance of the systems that recommend content and ads to our customers.
From optimizing engagement to measuring value
FAccT
Most recommendation engines today are based on predicting user engagement, e.g., predicting whether a user will click on an item or not. However, there is potentially a large gap between engagement signals and the desired notion of "value" worth optimizing for.
Authors:
Smitha Milli, Luca Belli, Moritz Hardt
Publication date: 7/30/2021
Tuning Word2vec for Large Scale Recommendation Systems
RecSys
Word2vec is a powerful machine learning tool that emerged from Natural Language Processing (NLP) and is now applied in multiple domains, including recommender systems, forecasting, and network analysis. As Word2vec is often used off the shelf, we address the question of whether the default hyperparameters are suitable for recommender systems.
Authors:
Benjamin P. Chamberlain, Emanuele Rossi, Dan Shiebler, Suvash Sedhain, Michael M. Bronstein
Publication date: 2020
Deep Bayesian Bandits: Exploring in Online Personalized Recommendations
RecSys
In this work, we formulate a display advertising recommender as a contextual bandit and implement exploration techniques that require sampling from the posterior distribution of click-through-rates in a computationally tractable manner.
Authors:
Dalin Guo, Sofia Ira Ktena, Pranay Kumar Myana, Ferenc Huszár, Wenzhe Shi, Alykhan Tejani, Michael Kneier, Sourav Das
Publication date: 2020
Model Size Reduction Using Frequency Based Double Hashing for Recommender Systems
RecSys
In this paper, we propose a hybrid hashing method to combine frequency hashing and double hashing techniques for model size reduction, without compromising performance.
Authors:
Caojin Zhang, Yicun Liu, Yuanpu Xie, Sofia Ira Ktena, Alykhan Tejani, Akshay Gupta, Pranay Kumar Myana, Deepak Dilipkumar, Suvadip Paul, Ikuhiro Ihara, Prasang Upadhyaya, Ferenc Huszár, Wenzhe Shi
Publication date: 2020
Doubly Robust Joint Learning for Recommendation on Data Missing Not at Random
ICML
First, we propose an estimator that integrates the imputed errors and propensities in a doubly robust way to obtain unbiased performance estimation and alleviate the effect of the propensity variance. Based on this estimator, we propose joint learning of rating prediction and error imputation to achieve good performance guarantees.
Authors:
Xiaojie Wang, Rui Zhang, Yu Sun, Jianzhong Qi
Publication date: 2019
A Correlation Maximization Approach for Cross Domain Co-Embeddings
arXiv
In this paper, we introduce ImplicitCE, an algorithm for recommending items to new users during their sign-up flow.
Authors:
Dan Shiebler
Publication date: 2018
RecService: Distributed Real-Time Graph Processing at Twitter-Embeddings
HotCloud
We present RecService, a distributed real-time graph processing engine that drives billions of recommendations on Twitter.
Authors:
Ajeet Grewal, Jerry Jiang, Gary Lam, Tristan Jung, Lohith Vuddemarri, Quannan Li, Aaditya Landge, Jimmy Lin
Publication date: 2018
A Meta-Learning Perspective on Cold-Start Recommendations for Items
NeurIPS
In this paper, we present a meta-learning strategy to address item cold-start when new items arrive continuously. We propose two deep neural network architectures that implement our meta-learning strategy.
Authors:
Manasi Vartak, Arvind Thiagarajan, Conrado Miranda, Jeshua Bratman, Hugo Larochelle
Publication date: 2017
GraphJet: Real-Time Content Recommendations at Twitter
HotCloud
This paper presents GraphJet, a new graph-based system for generating content recommendations at Twitter.
Authors:
Aneesh Sharma, Jerry Jiang, Praveen Bommannavar, Brian Larson, Jimmy Lin
Publication date: 2016
The Effect of Recommendations on Network Structure
WWW
Here we examine the aggregate effects of such recommendations on network structure, focusing on whether these recommendations increase the popularity of niche users or, conversely, those who are already popular.
Authors:
Jessica Su, Aneesh Sharma, Sharad Goel
Publication date: 2016
Real-time twitter recommendation: online motif detection in large dynamic graphs
VLDB Endorsement
We describe a production Twitter system for generating relevant, personalized, and timely recommendations based on observing the temporally-correlated actions of each user’s followings. The system currently serves millions of recommendations daily to tens of millions of mobile users.
Authors:
Pankaj Gupta, Venu Satuluri, Ajeet Grewal, Siva Gurumurthy, Volodymyr Zhabiuk, Quannan Li, Jimmy Lin
Publication date: 2014
User Modeling
Using machine learning to understand the behavior of users (e.g. what they’re interested in and how they interact).
An Experimental Study of Structural Diversity in Social Networks
ICWSM
We investigate the role of structural diversity on retention by conducting a large-scale randomized controlled study on the Twitter platform.
Authors:
Jessica Su, Krishna Kamath, Aneesh Sharma, Johan Ugander, Sharad Goel
Publication date: 2019
What is the Value of Experimentation and Measurement?
ICDM
We tackle this problem by analyzing how, by decreasing estimation uncertainty, E&M platforms allow for better prioritization. We quantify this benefit in terms of expected relative improvement in the performance of all new propositions and provide guidance for how much an E&M capability is worth and when organizations should invest in one.
Authors:
Bryan Liu, Ben Chamberlain
Publication date: 2019
Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR prediction
RecSys
The focus of this paper is to identify the best combination of loss functions and models that enable large-scale learning from a continuous stream of data in the presence of delayed labels. In this work, we compare 5 different loss functions, 3 of them applied to this problem for the first time.
Authors:
Sofia Ira Ktena, Alykhan Tejani, Lucas Theis, Pranay Kumar Myana, Deepak Dilipkumar, Ferenc Huszar, Steven Yoo, Wenzhe Shi
Publication date: 2019
Detecting Strong Ties Using Network Motifs
WWW
In this work, we demonstrate via experiments on Twitter data that using only such structural network features is sufficient for detecting strong ties with high precision. These structural network features are obtained from the presence and frequency of small network motifs on combined strong and weak ties.
Authors:
Rahmtin Rotabi, Krishna Kamath, Jon Kleinberg, Aneesh Sharma
Publication date: 2017
Cascades: a view from audience
WWW
Cascades on social and information networks have been a tremendously popular subject of study in the past decade, and there is a considerable literature on phenomena such as diffusion mechanisms, virality, cascade prediction, and peer network effects. Against the backdrop of this research, a basic question has received comparatively little attention: how desirable are cascades on a social media platform from the point of view of users?
Authors:
Rahmtin Rotabi, Krishna Kamath, Jon Kleinberg, Aneesh Sharma
Publication date: 2017
Graph Learning
Graph structured data naturally exists in many forms and at a scale from thousands to billions of nodes, leveraged for many product areas.
Understanding over-squashing and bottlenecks on graphs via curvature
ICLR
We provide a precise description of the over-squashing phenomenon in GNNs and analyze how it arises from bottlenecks in the graph.
Authors:
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, Michael M. Bronstein
Publication date: 3/16/2022
GRAND: Graph Neural Diffusion
ICML
We present Graph Neural Diffusion (GRAND) that approaches deep learning on graphs as a continuous diffusion process and treats Graph Neural Networks (GNNs) as discretisations of an underlying PDE.
Authors:
Ben Chamberlain, James Rowbottom, Maria Gorinova, Stefan Webb, Emanuele Rossi, Michael M. Bronstein
Publication date: 7/30/2021
Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networks
ICML
The pairwise interaction paradigm of graph machine learning has predominantly governed the modelling of relational systems. However, graphs alone cannot capture the multi-level interactions present in many complex systems and the expressive power of such schemes was proven to be limited.
Authors:
Cristian Bodnar, Fabrizio Frasca, Yu Guang Wang, Nina Otter, Guido Montúfar, Pietro Liò, Michael Bronstein
Publication date: 7/30/2021
Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
PMLR
This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020.
Authors:
Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, Isabelle Guyon
Publication date: 7/30/2021
Geometric Deep Learning
Book Preview
While learning generic functions in high dimensions is a cursed estimation problem, most tasks of interest are not generic, and come with essential pre-defined regularities arising from the underlying low-dimensionality and structure of the physical world. This text is concerned with exposing these regularities through unified geometric principles that can be applied throughout a wide spectrum of applications..
Authors:
Michael Bronstein, Joan Bruna, Taco Cohen, Petar Veličković
Publication date: 7/30/2021
Fast geometric learning with symbolic matrices
NeurIPS
Geometric methods rely on tensors that can be encoded using a symbolic formula and data arrays, such as kernel and distance matrices. We present an extension for standard machine learning frameworks that provides comprehensive support for this abstraction on CPUs and GPUs: our toolbox combines a versatile, transparent user interface with fast runtimes and low memory usage.
Authors:
Jean Feydy, Joan Alexis Glaunès, Benjamin Charlier, Michael M Bronstein
Publication date: 2020
SIGN scalable graph neural network architecture
ICML GRL+ Workshop, WWW
In this paper we propose a new, efficient and scalable graph deep learning architecture which sidesteps the need for graph sampling by using graph convolutional filters of different size that are amenable to efficient precomputation, allowing extremely fast training and inference.
Authors:
Fabrizio Frasca, Emanuele Rossi, Davide Eynard, Ben Chamberlain, Michael Bronstein, Federico Monti
Publication date: 2020
Temporal Graph Networks for deep learning on dynamic graphs
WWW
In this paper, we present Temporal Graph Networks (TGNs), a generic, efficient framework for deep learning on dynamic graphs represented as sequences of timed events.
Authors:
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, Michael Bronstein
Publication date: 2020
Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting
arXiv
We propose "Graph Substructure Networks" (GSN), a topologically-aware message passing scheme based on substructure encoding. We theoretically analyse the expressive power of our architecture, showing that it is strictly more expressive than the WL test, and provide sufficient conditions for universality.
Authors:
Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, Michael M. Bronstein
Publication date: 2020
Differentiable Graph Module
arXiv
In this paper, we introduce Differentiable Graph Module (DGM), a learnable function predicting the edge probability in the graph relevant for the task, that can be combined with convolutional graph neural network layers and trained in an end-to-end fashion.
Authors:
Anees Kazi, Luca Cosmo, Nassir Navab, Michael Bronstein
Publication date: 2020
Transferability of Spectral Graph Convolutional Neural Networks
arXiv
This paper focuses on spectral graph convolutional neural networks (ConvNets), where filters are defined as elementwise multiplication in the frequency domain of a graph.
Authors:
Ron Levie, Michael M. Bronstein, Gitta Kutyniok
Publication date: 2019
Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation
ICCV
In this paper, we focus on 3D deformable shapes that share a common topological structure, such as human faces and bodies.
Authors:
Giorgos Bouritsas, Sergiy Bokhnyak, Stylianos Ploumpis, Michael M. Bronstein, Stefanos Zafeiriou
Publication date: 2019
Single Image 3D Hand Reconstruction with Mesh Convolutions
BMVC
We evaluate the quality of the mesh reconstructions produced by the decoder on a new dataset and show latent space interpolation results.
Authors:
Dominik Kulon, Haoyang Wang, Riza Alp Güler, Michael M. Bronstein, Stefanos Zafeiriou
Publication date: 2019
ncRNA Classification with Graph Convolutional Networks
KDD DGL Workshop
We improve on the stateof-the-art for this task with a graph convolutional network model which achieves an accuracy of 85.73% and an F1-score of 85.61% over 13 classes.
Authors:
Emanuele Rossi, Federico Monti, Pietro Liò, Michael Bronstein
Publication date: 2019
Fake News Detection on Social Media using Geometric Deep Learning
ICLR RLG Workshop
In this paper, we show a novel automatic fake news detection model based on geometric deep learning.
Authors:
Federico Monti, Fabrizio Frasca, Davide Eynard, Damon Mannion, Michael Bronstein
Publication date: 2019
Faster gaze prediction with dense networks and Fisher pruning
arXiv
We first present a simple yet principled greedy pruning method which we call Fisher pruning. Through a combination of knowledge distillation and Fisher pruning, we obtain much more runtime-efficient architectures for saliency prediction, achieving a 10x speedup for the same AUC performance as a state of the art network on the CAT2000 dataset.
Authors:
Lucas Theis, Iryna Korshunova, Alykhan Tejani, Ferenc Huszár
Publication date: 2018
Frame Interpolation with Multi-Scale Deep Loss Functions and Generative Adversarial Networks
arXiv
To maximise the efficiency of our network, we propose a novel multiscale residual estimation module where the predicted flow and synthesised frame are constructed in a coarse-to-fine fashion.
Authors:
Joost van Amersfoort, Wenzhe Shi, Alejandro Acosta, Francisco Massa, Johannes Totz, Zehan Wang, Jose Caballero
Publication date: 2018
KDGAN: Knowledge Distillation with Generative Adversarial Networks
NeurIPS
We propose a three-player game named KDGAN consisting of a classifier, a teacher, and a discriminator. The classifier and the teacher learn from each other via distillation losses and are adversarially trained against the discriminator via adversarial losses.
Authors:
Xiaojie Wang, Rui Zhang, Yu Sun, Jianzhong Qi
Publication date: 2018
Realgraph: User interaction prediction at Twitter
User Engagement Optimization
In this work, we provide a framework to compute relationship strength for ties based on directed interactions between users. The proposed framework, called RealGraph, produces a directed and weighted graph where the nodes are Twitter users, and the edges are labeled with interactions between a directed pair of users.
Authors:
Krishna Kamath, Aneesh Sharma, Dong Wang, Zhijun Yin
Publication date: 2014
Machine Learning Ethics Research
Research, development of best practices, and tools to advance the responsible use of algorithms, proactive risk assessments and mitigations.
Algorithmic Amplification of Politics on Twitter
Our results reveal a remarkably consistent trend: In 6 out of 7 countries studied, the mainstream political right enjoys higher algorithmic amplification than the mainstream political left. Consistent with this overall trend, our second set of findings studying the U.S. media landscape revealed that algorithmic amplification favours right-leaning news sources.
Authors:
Ferenc Huszár, Sofia Ira Ktena, Conor O’Brien, Luca Belli, Andrew Schlaikjera, and Moritz Hardt
Publication date: October 2021
Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency
CSWM
Twitter uses machine learning to crop images, where crops are centered around the part predicted to be the most salient. In fall 2020, Twitter users raised concerns that the automated image cropping system on Twitter favored light-skinned over dark-skinned individuals, as well as concerns that the system favored cropping woman's bodies instead of their heads. In order to address these concerns, we conduct an extensive analysis using formalized group fairness metrics.
Authors:
Kyra Yee, Uthaipon Tantipongpipat, Shubhanshu Mishra
Paper received an Impact Recognition award at CSCW 2021.
Publication date: 5/18/2021
Lessons Learned Addressing Dataset Bias in Model-Based Candidate Generation at Twitter
ISR-KDD workshop
We first explore the dynamics of the dataset bias problem and then demonstrate how to use random sampling techniques to mitigate it. Finally, in a novel application of fine-tuning, we show performance gains when applying our candidate generation system to Twitter's home timeline.
Authors:
Alim Virani, Jay Baxter, Dan Shiebler, Philip Gautier, Shivam Verma, Yan Xia, Apoorv Sharma, Sumit Binnani, Linlin Chen, Chenguang Yu
Publication date: 2020
Assessing Demographic Bias in Named Entity Recognition
KG-BIAS workshop
In this work, we assess the bias in various Named Entity Recognition (NER) systems for English across different demographic groups with synthetically generated corpora
Authors:
Shubhanshu Mishra, Sijun He, Luca Belli
Publication date: 2020
Other Publications
Additional applied research and publications that don’t fall into the above categories and topics can be found here.
Isotonic Regression Adjustment for Variance Reduction
CODE@MIT
This paper presents a novel approach for variance reduction in A/B experiments that is based on isotonic regression, which works as well as more complicated methods but is computationally much more efficient.
Authors:
Ryan Turner, Umashanthi Pavalanathan, Stefan Webb, Nils Hammerla, Brent Cohn, Anthony Fu
Publication date: 5/11/2021
Context-Uncertainty-Aware Chatbot Action Selection via Parameterized Auxiliary Reinforcement Learning
PAKDD
We propose a context-uncertainty-aware chatbot and a reinforcement learning (RL) model to train the chatbot. The proposed model is named Parameterized Auxiliary Asynchronous Advantage Actor Critic (PA4C)
Authors:
Chuandong Yin, Rui Zhang, Jianzhong Qi, Yu Sun, Tenglun Tan
Publication date: 2018
Lossy Image Compression with Compressive Autoencoders
ICLR RLG Workshop
We propose a new approach to the problem of optimizing autoencoders for lossy image compression.
Authors:
Lucas Theis, Wenzhe Shi, Andrew Cunningham, Ferenc Huszár
Publication date: 2017
Fast Face-swap Using Convolutional Neural Networks
ICCV
By combining neural networks with simple pre- and post-processing steps, we aim at making face swap work in real-time with no input from the user.
Authors:
Iryna Korshunova, Wenzhe Shi, Joni Dambre, Lucas Theis
Publication date: 2017
Variational Inference using Implicit Distributions
arXiv
This paper provides a unifying review of existing algorithms establishing connections between variational autoencoders, adversarially learned inference, operator VI, GAN-based image reconstruction, and more. Secondly, the paper provides a framework for building new algorithms: depending on the way the variational bound is expressed we introduce prior-contrastive and jointcontrastive methods, and show practical inference algorithms based on either density ratio estimation or denoising.
Authors:
Ferenc Huszár
Publication date: 2017
Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize
arXiv
A note on sub-pixel convolution, resize convolution and convolution resize
Authors:
Andrew Aitken, Christian Ledig, Lucas Theis, Jose Caballero, Zehan Wang, Wenzhe Shi
Publication date: 2017
Amortised MAP Inference for Image Super-resolution
ICLR
We propose three methods to solve this optimisation problem: (1) Generative Adversarial Networks (GAN) (2) denoiser-guided SR which backpropagates gradient-estimates from denoising to train the network, and (3) a baseline method using a maximum-likelihoodtrained image prior. Our experiments show that the GAN based approach performs best on real image data. Lastly, we establish a connection between GANs and amortised variational inference as in e. g. variational autoencoders.
Authors:
Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, Ferenc Huszár
Publication date: 2017
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
CVPR
In this paper, we present SRGAN, a generative adversarial network (GAN) for image superresolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4× upscaling factors.
Authors:
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi
Publication date: 2017
Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation
CVPR
In this paper, we introduce spatio-temporal subpixel convolution networks that effectively exploit temporal redundancies and improve reconstruction accuracy while maintaining real-time speed. Specifically, we discuss the use of early fusion, slow fusion and 3D convolutions for the joint processing of multiple consecutive video frames. We also propose a novel joint motion compensation and video super-resolution algorithm that is orders of magnitude more efficient than competing methods, relying on a fast multi-resolution spatial transformer module that is endto-end trainable. Authors:
Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, Wenzhe Shi
Publication date: 2017
Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
CVPR
In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space.
Authors:
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, Zehan Wang
Publication date: 2016