Arun Mallya

Currently a Founding Member of Technical Staff at Warp Sciences working on robotics for science.

Previously a Research Scientist at Meta (Gen AI → MSL) working on video generation, video editing models, and model auto-evaluation. Previously a Senior Research Scientist in the Deep Imagination Research (DIR) group at NVIDIA (now Cosmos Lab). Part of the lab since its inception, when it began with just 3 members.

PhD from the University of Illinois at Urbana-Champaign, advised by Prof. Svetlana Lazebnik. MS in CS from UIUC; B.Tech in CSE from IIT Kharagpur.

Prior to robotics, I worked on methods for generating images and videos as well as their evaluation with vision-language models. I also dabbled in neural rendering and neural network efficiency, as well as probing their unexpected properties. My work on facial animation received the Best in Show award at the 2021 SIGGRAPH Real-time Live!, a showcase of cutting-edge, real-time technology demos.

LinkedIn Scholar GitHub Webpage
Arun Mallya

Selected Research

See all on Google Scholar →

Image/Video Generation

Muse Image/Video Generation Models
Image and video generation models deployed in Meta AI and Instagram
Website
Video editing, released in Meta AI
Video-editing diffusion models, <15s latency for 1080p 10s video editing
Tweet Website
Movie Gen: A Cast of Media Foundation Models
Technical Report, Meta, 2024
Text-to-video, video-editing diffusion models
arXiv Project Page
Edify Image teaser
Edify Image Generation
Technical Report, NVIDIA, 2024
Image generation with pixel-diffusion models
arXiv Project Page

Facial Animation

SPACE teaser
SPACE: Speech-driven Portrait Animation with Controllable Expression
Siddharth Gururani, Arun Mallya, Ting-Chun Wang, Rafael Valle, Ming-Yu Liu
International Conference on Computer Vision (ICCV), 2023
Facial animation with lip-sync using just input speech
arXiv Project Page
IMWA teaser
Implicit Warping for Animation with Image Sets
Arun Mallya, Ting-Chun Wang, Ming-Yu Liu
Neural Information Processing Systems (NeurIPS), 2022
Complex motion transfer warping replaced by a single cross-attention layer
arXiv Project Page
face-vid2vid teaser
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing oral
Ting-Chun Wang, Arun Mallya, Ming-Yu Liu
Computer Vision and Pattern Recognition (CVPR), 2021
3-D controllable facial animation using motion transfer from video
arXiv Project Page

Neural Rendering

LoE teaser
Implicit Neural Representations with Levels-of-Experts
Zekun Hao, Arun Mallya, Serge Belongie, Ming-Yu Liu
Neural Information Processing Systems (NeurIPS), 2022
Like MoE, but using a hierarchy of position-dependent and periodic weights
Paper
GANcraft teaser
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds oral
Zekun Hao, Arun Mallya, Serge Belongie, Ming-Yu Liu
International Conference on Computer Vision (ICCV), 2021
Generating realistic view-consistent renders of 3D block worlds
arXiv Project Page & Code

Model Efficiency / Interesting Properties

GradInversion teaser
See through Gradients: Image Batch Recovery via GradInversion
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose Alvarez, Pavlo Molchanov, Jan Kautz
Computer Vision and Pattern Recognition (CVPR), 2021
Invert an entire batch of training images just from averaged gradients
Paper
DeepInversion teaser
Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion oral
Hongxu Yin, Pavlo Molchanov, Zhizhong Li, Jose Alvarez, Arun Mallya, Derek Hoiem, Niraj Jha, Jan Kautz
Computer Vision and Pattern Recognition (CVPR), 2020
Reproduce training data samples just from a trained neural network
arXiv Code
Piggyback teaser
Piggyback: Adding Multiple Tasks to a Single, Fixed Network by Learning to Mask
Arun Mallya, Dillon Davis, Svetlana Lazebnik
European Conference on Computer Vision (ECCV), 2018
Make a neural network learn new tasks by simply masking existing, fixed weights
arXiv Code
PackNet teaser
PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning
Arun Mallya, Svetlana Lazebnik
Computer Vision and Pattern Recognition (CVPR), 2018
Add new tasks to a neural network by pruning and training a small number of weights
arXiv Code

Tutorials & Workshops

  1. Machine Learning with Synthetic Data, CVPR 2022
  2. Accelerating Computer Vision with Mixed Precision, ECCV 2020
  3. Accelerating Computer Vision with Mixed Precision, ICCV 2019

Writeups & Notes

Hosted on GitHub. Edit requests, additions, and corrections are welcome.

  1. A Backpropagation Refresher
  2. An Illustrated Explanation of the LSTM Forward-Backward Pass
  3. Introduction to RNNs
  4. Introduction to RNNs — II
  5. Jupyter notebook to find Receptive Field Size and Effective Stride (supports dilated convs)
  6. Visualization of neuron connections and receptive field of a CNN (including dilation)