I am a Research Scientist at Apple AI/ML working with Jeff Nichols. My research focuses on developing multi-modal LLMs and their UI and agentic applications. Previously I was a Research Scientist at Google Research working on generative models for creative and design applications. I am based in Hong Kong | Mountain View, CA.

I recently graduated with a Ph.D. in Computer Science from University of California, Berkeley advised by Prof. John Canny.

descriptionCV (Feb 2022)

rss_feed news

  • 12/11/2023 I recently joined Apple AI/ML, working with Jeff Nichols starting Winter 2023.

  • 07/25/2022 I graduated from UC Berkeley with a Ph.D. in EECS! I am excited to be joining Google Research working with Yang Li starting Fall 2022.

  • 07/27/2022 Our Paper Enabling Multi-modal Search for Inspirational Design Stimuli using Deep Learning is published in the AI EDAM Journal.

  • 04/30/2022 Our Workshop Paper Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models is published at the Computational UI Workshop at CHI 2022.

  • 11/20/2019 I am excited to receive an Honourable Mention for the Adobe Research Fellowship 2020.

library_books publications

(note: publications prior to 2018 published as Zifeng Huang)

Book Chapters

  • Sketch-based Creativity Support Tools using Deep Learning
    Forrest Huang, Eldon Schoop, David Ha, Jeffrey Nichols, and John Canny
    In Artificial Intelligence for Human Copmuter Interaction: A Modern Approach (Springer 2021) | Preprint | Book

  • An Early Rico Retrospective: Three Years of Uses for a Mobile App Dataset
    Biplab Deka, Bardia Doosti, Forrest Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Ranjitha Kumar, Tao Dong, and Jeffrey Nichols
    In Artificial Intelligence for Human Copmuter Interaction: A Modern Approach (Springer 2021) | Book


Peer-reviewed Conference Publications

  • MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
    Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang (Core Author), Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, Sam Dodge, Keen You, Zhen Yang, Aleksei Timofeev, Mingze Xu, Hong-You Chen, Jean-Philippe Fauconnier, Zhengfeng Lai, Haoxuan You, Zirui Wang, Afshin Dehghan, Peter Grasch, and Yinfei Yang
    Proceedings of ICLR '25 | Paper (Preprint)

  • Automatic Macro Mining from Interaction Traces at Scale
    Forrest Huang, Gang Li, Tao Li, and Yang Li
    Proceedings of CHI '24 | Paper (Preprint)

  • PLay: Parametrically Conditioned Layout Generation using Latent Diffusion
    Chin-Yi Cheng, Forrest Huang, Gang Li, and Yang Li
    Proceedings of ICML '23 | Paper

  • Multi-modal Search for Inspirational Examples in Design
    Elisa Kwon, Forrest Huang, and Kosa Goucher-Lambert
    Proceedings of IDETC '21 | Paper

  • UMLAUT: Debugging Deep Learning Programs using Program Structure and Model Behavior
    Eldon Schoop, Forrest Huang, and Björn Hartmann
    Proceedings of CHI '21 | Paper

  • Scones: Towards Conversational Authoring of Sketches
    Forrest Huang, Eldon Schoop, David Ha, and John F. Canny
    Proceedings of IUI '20 (Long Paper) | Paper

  • Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications
    Forrest Huang, and John F. Canny
    Proceedings of UIST '19 | Paper

  • Swire: Sketch-based User Interface Retrieval
    Forrest Huang, John F. Canny, and Jeffrey Nichols
    Proceedings of CHI '19 | Paper

  • MakerLens: What Sign-In, Reservation and Training Data Can (and Cannot) Tell You About Your Makerspace
    Eldon Schoop, Forrest Huang, Nathan Khuu, and Björn Hartmann
    Proceedings of ISAM '18 | Paper

  • ZIPT: Zero-Integration Performance Testing of Mobile App Designs
    Biplab Deka, Zifeng Huang, Chad Franzen, Jeffrey Nichols, Yang Li, and Ranjitha Kumar
    Proceedings of UIST '17 | Paper

  • Rico: A Mobile App Dataset for Building Data-Driven Design Applications
    Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar
    Proceedings of UIST '17 | Paper

  • ERICA: Interaction Mining for Mobile Apps
    Biplab Deka, Zifeng Huang, and Ranjitha Kumar
    Proceedings of UIST '16 | Paper


Journal Articles

  • Enabling Multi-modal Search for Inspirational Design Stimuli using Deep Learning
    Elisa Kwon, Forrest Huang, and Kosa Goucher-Lambert
    AI EDAM '22 | Paper

  • GPU-Accelerated t-Distributed Stochastic Neighbor Embedding
    David M. Chan*, Roshan Rao*, Forrest Huang*, and John F. Canny
    Journal of Parallel and Distributed Computing | Paper


Workshop Papers | Posters | Extended Abstracts

  • Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models
    Forrest Huang, Gang Li, Xin Zhou, John F. Canny, and Yang Li
    CHI 2022 Computational UI Workshop | Paper

  • SCRAM: Simple Checks for Realtime Analysis of Model Training for Non-Expert ML Programmers
    Eldon Schoop, Forrest Huang, and Björn Hartmann
    Late-Breaking Works of CHI '20 and ICML 2020 Workshop on Human in the Loop Learning | LBW

  • t-SNE-CUDA: GPU-Accelerated t-SNE and its Applications to Modern Data
    David M. Chan*, Roshan Rao*, Forrest Huang*, and John F. Canny
    HPML Workshop '18 Outstanding Paper Award | Paper

  • Ranking Designs and Users in Online Social Networks
    Biplab Deka, Haizi Yu, Devin Ho, Zifeng Huang, Jerry O. Talton, and Ranjitha Kumar
    Extended Abstracts of CHI '15 | Poster | Extended Abstract