Publications

2023

1. ICLR

   Causal Balancing for Domain Generalization

   Wang, Xinyi, Saxon, Michael, Li, Jiachen, Zhang, Hongyang, Zhang, Kun, and Wang, William Yang

   In International Conference on Learning Representations 2023
2. ICLR

   Tier Balancing: Towards Dynamic Fairness over Underlying Causal Factors

   Tang, Zeyu, Chen, Yatong, Liu, Yang, and Zhang, Kun

   In The Eleventh International Conference on Learning Representations 2023
3. ICLR

   Calibration Matters: Tackling Maximization Bias in Large-scale Advertising Recommendation Systems

   Fan, Yewen, Si, Nian, and Zhang, Kun

   In International Conference on Learning Representations 2023
4. ICLR

   GAIN: On the Generalization of Instructional Action Understanding

   Li, Junlong, Chen, Guangyi, Tang, Yansong, Bao, Jinan, Zhang, Kun, Zhou, Jie, and Lu, Jiwen

   In International Conference on Learning Representations 2023
5. ICLR

   Scalable Estimation of Nonparametric Markov Networks with Mixed-Type Data

   Zheng, Yujia, Ng, Ignavier, Fan, Yewen, and Zhang, Kun

   In International Conference on Learning Representations 2023
6. ICML

   Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach

   Guo, Han, Greengard, Philip, Wang, Hongyi, Gelman, Andrew, Xing, Eric, and Kim, Yoon

   In International Conference on Learning Representations 2023
7. arXiv

   Does compressing activations help model parallel training?

   Bian, Song, Li, Dacheng, Wang, Hongyi, Xing, Eric P., and Venkataraman, Shivaram

   2023

2022

1. NeurIPS

   Factored Adaptation for Non-Stationary Reinforcement Learning

   Feng, Fan, Huang, Biwei, Zhang, Kun, and Magliacane, Sara

   In Advances in Neural Information Processing Systems 2022
2. Identification of Linear Non-Gaussian Latent Hierarchical Structure

   Xie, Feng, Huang, Biwei, Chen, Zhengming, He, Yangbo, Geng, Zhi, and Zhang, Kun

   In Proceedings of the 39th International Conference on Machine Learning 17–23 jul 2022

   Abs PDF

   Traditional causal discovery methods mainly focus on estimating causal relations among measured variables, but in many real-world problems, such as questionnaire-based psychometric studies, measured variables are generated by latent variables that are causally related. Accordingly, this paper investigates the problem of discovering the hidden causal variables and estimating the causal structure, including both the causal relations among latent variables and those between latent and measured variables. We relax the frequently-used measurement assumption and allow the children of latent variables to be latent as well, and hence deal with a specific type of latent hierarchical causal structure. In particular, we define a minimal latent hierarchical structure and show that for linear non-Gaussian models with the minimal latent hierarchical structure, the whole structure is identifiable from only the measured variables. Moreover, we develop a principled method to identify the structure by testing for Generalized Independent Noise (GIN) conditions in specific ways. Experimental results on both synthetic and real-world data show the effectiveness of the proposed approach.

3. NeurIPS

   Scalable Causal Discovery with Score Matching

   Montagna, Francesco, Noceti, Nicoletta, Rosasco, Lorenzo, Zhang, Kun, and Locatello, Francesco

   In NeurIPS 2022 Workshop on Score-Based Methods 2022
4. arXiv

   Prompt Learning with Optimal Transport for Vision-Language Models

   Chen, Guangyi, Yao, Weiran, Song, Xiangchen, Li, Xinyue, Rao, Yongming, and Zhang, Kun

   2022
5. ICML

   Partial disentanglement for domain adaptation

   Kong, Lingjing, Xie, Shaoan, Yao, Weiran, Zheng, Yujia, Chen, Guangyi, Stojanov, Petar, Akinwande, Victor, and Zhang, Kun

   In Proceedings of the 39th International Conference on Machine Learning 17–23 jul 2022

   Abs PDF

   Unsupervised domain adaptation is critical to many real-world applications where label information is unavailable in the target domain. In general, without further assumptions, the joint distribution of the features and the label is not identifiable in the target domain. To address this issue, we rely on a property of minimal changes of causal mechanisms across domains to minimize unnecessary influences of domain shift. To encode this property, we first formulate the data generating process using a latent variable model with two partitioned latent subspaces: invariant components whose distributions stay the same across domains, and sparse changing components that vary across domains. We further constrain the domain shift to have a restrictive influence on the changing components. Under mild conditions, we show that the latent variables are partially identifiable, from which it follows that the joint distribution of data and labels in the target domain is also identifiable. Given the theoretical insights, we propose a practical domain adaptation framework, called iMSDA. Extensive experimental results reveal that iMSDA outperforms state-of-the-art domain adaptation algorithms on benchmark datasets, demonstrating the effectiveness of our framework.

6. PMLR

   On the Convergence of Continuous Constrained Optimization for Structure Learning

   Ng, Ignavier, Lachapelle, Sebastien, Rosemary Ke, Nan, Lacoste-Julien, Simon, and Zhang, Kun

   In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics 28–30 mar 2022

   Abs PDF

   Recently, structure learning of directed acyclic graphs (DAGs) has been formulated as a continuous optimization problem by leveraging an algebraic characterization of acyclicity. The constrained problem is solved using the augmented Lagrangian method (ALM) which is often preferred to the quadratic penalty method (QPM) by virtue of its standard convergence result that does not require the penalty coefficient to go to infinity, hence avoiding ill-conditioning. However, the convergence properties of these methods for structure learning, including whether they are guaranteed to return a DAG solution, remain unclear, which might limit their practical applications. In this work, we examine the convergence of ALM and QPM for structure learning in the linear, nonlinear, and confounded cases. We show that the standard convergence result of ALM does not hold in these settings, and demonstrate empirically that its behavior is akin to that of the QPM which is prone to ill-conditioning. We further establish the convergence guarantee of QPM to a DAG solution, under mild conditions. Lastly, we connect our theoretical results with existing approaches to help resolve the convergence issue, and verify our findings in light of an empirical comparison of them.

7. arXiv

   The Impact of Symbolic Representations on In-context Learning for Few-shot Reasoning

   Zhang, Hanlin, Zhang, Yi-Fan, Li, Li Erran, and Xing, Eric

   2022
8. JCB

   Kernel Mixed Model for Transcriptome Association Study

   Wang, Haohan, Lopez, Oscar, Xing, Eric P., and Wu, Wei

   Journal of Computational Biology 2022
9. arXiv

   Expeditious Saliency-guided Mix-up through Random Gradient Thresholding

   Luu, Minh-Long, Huang, Zeyi, Xing, Eric P., Lee, Yong Jae, and Wang, Haohan

   2022
10. arXiv

    On Optimizing the Communication of Model Parallelism

    Zhuang, Yonghao, Zhao, Hexu, Zheng, Lianmin, Li, Zhuohan, Xing, Eric P., Ho, Qirong, Gonzalez, Joseph E., Stoica, Ion, and Zhang, Hao

    2022
11. ECCV

    A Fast Knowledge Distillation Framework for Visual Recognition

    Shen, Zhiqiang, and Xing, Eric

    In Computer Vision – ECCV 2022 2022

    Abs

    While Knowledge Distillation (KD) has been recognized as a useful tool in many visual tasks, such as supervised classification and self-supervised representation learning, the main drawback of a vanilla KD framework is its mechanism that consumes the majority of the computational overhead on forwarding through the giant teacher networks, making the entire learning procedure inefficient and costly. The recently proposed solution ReLabel suggests creating a label map for the entire image. During training, it receives the cropped region-level label by RoI aligning on a pre-generated entire label map, which allows for efficient supervision generation without having to pass through the teachers repeatedly. However, as the pre-trained teacher employed in ReLabel is from the conventional multi-crop scheme, there are various mismatches between the global label-map and region-level labels in this technique, resulting in performance deterioration compared to the vanilla KD. In this study, we present a Fast Knowledge Distillation (FKD) framework that replicates the distillation training phase and generates soft labels using the multi-crop KD approach, meanwhile training faster than ReLabel since no post-processes such as RoI align and softmax operations are used. When conducting multi-crop in the same image for data loading, our FKD is even more efficient than the traditional image classification framework. On ImageNet-1K, we obtain 80.1% Top-1 accuracy on ResNet-50, outperforming ReLabel by 1.2% while being faster in training and more flexible to use. On the distillation-based self-supervised learning task, we also show that FKD has an efficiency advantage.

12. ECCV

    Sliced Recursive Transformer

    Shen, Zhiqiang, Liu, Zechun, and Xing, Eric

    In Computer Vision – ECCV 2022 2022

    Abs

    We present a neat yet effective recursive operation on vision transformers that can improve parameter utilization without involving additional parameters. This is achieved by sharing weights across depth of transformer networks. The proposed method can obtain a substantial gain (}}\backslashsim }}∼2%) simply using naïve recursive operation, requires no special or sophisticated knowledge for designing principles of networks, and introduces minimal computational overhead to the training procedure. To reduce the additional computation caused by recursive operation while maintaining the superior accuracy, we propose an approximating method through multiple sliced group self-attentions across recursive layers which can reduce the cost consumption by 10–30% without sacrificing performance. We call our model Sliced Recursive Transformer (SReT), a novel and parameter-efficient vision transformer design that is compatible with a broad range of other designs for efficient ViT architectures. Our best model establishes significant improvement on ImageNet-1K over state-of-the-art methods while containing fewer parameters. The proposed weight sharing mechanism by sliced recursion structure allows us to build a transformer with more than 100 or even 1000 shared layers with ease while keeping a compact size (13–15 M), to avoid optimization difficulties when the model is too large. The flexible scalability has shown great potential for scaling up models and constructing extremely deep vision transformers. Code is available at https://github.com/szq0214/SReT.

13. TPAMI

    Meta-DETR: Image-Level Few-Shot Detection with Inter-Class Correlation Exploitation

    Zhang, Gongjie, Luo, Zhipeng, Cui, Kaiwen, Lu, Shijian, and Xing, Eric P.

    IEEE Transactions on Pattern Analysis and Machine Intelligence 2022
14. TNNLS

    Prototypical Graph Contrastive Learning

    Lin, Shuai, Liu, Chen, Zhou, Pan, Hu, Zi-Yuan, Wang, Shuojia, Zhao, Ruihui, Zheng, Yefeng, Lin, Liang, Xing, Eric, and Liang, Xiaodan

    IEEE Transactions on Neural Networks and Learning Systems 2022
15. OSDI

    Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

    Zheng, Lianmin, Li, Zhuohan, Zhang, Hao, Zhuang, Yonghao, Chen, Zhifeng, Huang, Yanping, Wang, Yida, Xu, Yuanzhong, Zhuo, Danyang, Xing, Eric P., Gonzalez, Joseph E., and Stoica, Ion

    In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) Jul 2022
16. CVPR

    Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space

    Chavan, Arnav, Shen, Zhiqiang, Liu, Zhuang, Liu, Zechun, Cheng, Kwang-Ting, and Xing, Eric P.

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Jun 2022
17. NeurIPS

    DAMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness

    Li, Dacheng, Wang, Hongyi, Xing, Eric P, and Zhang, Hao

    In Conference on Neural Information Processing Systems 2022
18. NeurIPS

    Rare Gems: Finding Lottery Tickets at Initialization

    Sreenivasan, Kartik, Sohn, Jy-yong Sohn, Yang, Liu, Grinde, Matthew, Nagle, Alliot, Wang, Hongyi, Xing, Eric P, Lee, Kangwook, and Papailiopoulos, Dimitris

    In Conference on Neural Information Processing Systems 2022
19. NeurIPS

    Masked Generative Adversarial Networks are Robust Generation Learners

    Huang, Jiaxing, Cui, Kaiwen, Guan, Dayan, Xiao, Aoran, Zhan, Fangneng, Lu, Shijian, Liao, Shengcai, and Xing, Eric P

    In Conference on Neural Information Processing Systems 2022
20. NeurIPS

    Uncovering the Structural Fairness in Graph Contrastive Learning

    Wang, Ruijia, Wang, Xiao, Shi, Chuan, and Song, Le

    In Conference on Neural Information Processing Systems 2022
21. NeurIPS

    On the Identifiability of Nonlinear ICA: Sparsity and Beyond

    Zheng, Yujia, Ng, Ignavier, and Zhang, Kun

    In Conference on Neural Information Processing Systems 2022
22. NeurIPS

    Independence Testing-Based Approach to Causal Discovery under Measurement Error and Linear Non-Gaussian Models

    Dai, Haoyue, Spirtes, Peter, and Zhang, Kun

    In Conference on Neural Information Processing Systems 2022
23. NeurIPS

    Temporal Disentangled Representation Learning

    Yao, Weiran, Chen, Guangyi, and Zhang, Kun

    In Conference on Neural Information Processing Systems 2022
24. NeurIPS

    Unsupervised Image-to-Image Translation with Density Changing Regularization

    Xie, Shaoan, Ho, Qirong, and Zhang, Kun

    In Conference on Neural Information Processing Systems 2022
25. NeurIPS

    Latent Hierarchical Causal Structure Discovery with Rank Constraints

    Huang, Biwei, Low, Charles, Xie, Feng, Glymour, Clark, and Zhang, Kun

    In Conference on Neural Information Processing Systems 2022
26. NeurIPS

    Factored Adaptation for Non-Stationary Reinforcement Learning

    Feng, Fan, Huang, Biwei, Zhang, Kun, and Magliacane, Sara

    In Conference on Neural Information Processing Systems 2022
27. NeurIPS

    Causal Discovery in Linear Latent Variable Models Subject to Measurement Error

    Yang, Yuqin, Ghassami, AmirEmad, Nafea, Mohamed S, Kiyavash, Negar, Zhang, Kun, and Shpitser, Ilya

    In Conference on Neural Information Processing Systems 2022
28. NeurIPS

    Counterfactual Fairness with Partially Known Causal Graph

    Zuo, Aoqi, Wei, Susan, Liu, Tongliang, Han, Bo, Zhang, Kun, and Gong, Mingming

    In Conference on Neural Information Processing Systems 2022
29. NeurIPS

    DMissDAG: Causal Discovery in the Presence of Missing Data with Continuous Additive Noise Models

    Gao, Erdun, Ng, Ignavier, Gong, Mingming, Shen, Li, Huang, Wei, Liu, Tongliang, Zhang, Kun, and Bondell, Howard

    In Conference on Neural Information Processing Systems 2022
30. NeurIPS

    Truncated Matrix Power Iteration for Differentiable DAG Learning

    Zhang, Zhen, Ng, Ignavier, Gong, Dong, Liu, Yuhang, Abbasnejad, Ehsan M, Gong, Mingming, Zhang, Kun, and Shi, Javen Qinfeng

    In Conference on Neural Information Processing Systems 2022
31. ICML

    Action-Sufficient State Representation Learning for Control with Structural Constraints

    Huang, Biwei, Lu, Chaochao, Leqi, Liu, Hernández-Lobato, José Miguel, Glymour, Clark, Schölkopf, Bernhard, and Zhang, Kun

    In International Conference on Machine Learning 2022
32. ICLR

    AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning

    Huang, Biwei, Feng, Fan, Lu, Chaochao, Magliacane, Sara, and Zhang, Kun

    In International Conference on Learning Representations (Spotlight) 2022
33. ICLR

    Learning Temporally Latent Causal Processes from General Temporal Data

    Yao, Weiran, Sun, Yuewen, Ho, Alex, Sun, Changyin, and Zhang, Kun

    In International Conference on Learning Representations 2022

2021

1. arXiv

   Panoramic Learning with A Standardized Machine Learning Formalism

   Hu, Zhiting, and Xing, Eric P

   arXiv preprint arXiv:2108.07783 2021
2. OSDI

   Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning

   Qiao, Aurick, Choe, Sang Keun, Subramanya, Suhas Jayaram, Neiswanger, Willie, Ho, Qirong, Zhang, Hao, Ganger, Gregory R, and Xing, Eric P

   In 15th {USENIX} Symposium on Operating Systems Design and Implementation (the Jay Lepreau Best Paper Award) 2021
3. NeurIPS

   Identification of partially observed linear causal models: Graphical conditions for the non-gaussian and heterogeneous cases

   Adams, Jeffrey, Hansen, Niels, and Zhang, Kun

   In Conference on Neural Information Processing Systems 2021
4. ICCV

   Unaligned image-to-image translation by learning to reweight

   Xie, Shaoan, Gong, Mingming, Xu, Yanwu, and Zhang, Kun

   In Proceedings of the International Conference on Computer Vision 2021

2020

1. ICLR

   Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms

   Al-Shedivat, Maruan, Gillenwater, Jennifer, Xing, Eric, and Rostamizadeh, Afshin

   In International Conference on Learning Representations 2020
2. JMLR

   Contextual Explanation Networks.

   Al-Shedivat, Maruan, Dubey, Avinava, and Xing, Eric P

   Journal of Machine Learning Research 2020
3. JMLR

   Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly.

   Kandasamy, Kirthevasan, Vysyaraju, Karun Raju, Neiswanger, Willie, Paria, Biswajit, Collins, Christopher R, Schneider, Jeff, Poczos, Barnabas, and Xing, Eric P

   Journal of Machine Learning Research 2020
4. NeurIPS

   Domain adaptation as a problem of inference on graphical models

   Zhang, Kun, Gong, Mingming, Stojanov, Petar, Huang, Biwei, Liu, Qingsong, and Glymour, Clark

   In Conference on Neural Information Processing Systems 2020
5. NeurIPS

   Generalized independent noise condition for estimating latent variable causal graphs

   Xie, Feng, Cai, Ruichu, Huang, Biwei, Glymour, Clark, Hao, Zhifeng, and Zhang, Kun

   In Conference on Neural Information Processing Systems (Spotlight) 2020
6. JMLR

   Causal Discovery from Heterogeneous/Nonstationary Data.

   Huang, Biwei, Zhang, Kun, Zhang, Jiji, Ramsey, Joseph D, Sanchez-Romero, Ruben, Glymour, Clark, and Schölkopf, Bernhard

   Journal of Machine Learning Research 2020