quick links

publications activities
bio
code
teaching

contact


Roland Memisevic

I am interested in AI systems that can see, understand and interact with humans.

I believe that the key to more human-like AI is enabling it to hold an audio-visual dialogue with real people, in real-time, about the real world.

I received the PhD from the University of Toronto in 2008 working with Geoffrey Hinton.

email: memisevr[at]iro[dot]umontreal[dot]ca


Bio, CV
  • This recent slide deck (Oct 2020) describes our work at Twenty Billion Neurons, a company I co-founded in 2015

  • Check out 20BN Sense, our open source software that can turn an input stream of pixels into gestures, actions, and other human behaviours

  • We've built an end-to-end AI-powered fitness companion as a show-case application

  • Demo videos showing our situated avatars based on the same technology

In the news

  • CNBC published an article about our AI fitness coach

  • My interview at the Chris Collin's show

  • Bloomberg wrote about one of our first AI avatar demos (this was at NeurIPS)

Publications

    2022

  • 2022 Memisevic, R.
    Metaphors we learn by
    [pdf]

  • 2022 Memisevic, R., Panchal, S., Lee, M.
    Decision Making as Language Generation
    NeurIPS Foundation Models for Decision Making workshop 2022
    [pdf]
  • 2020

  • 2020 Mercier, A., Berger, G., Panchal, S., Letsch, F., Boehm, C., Bax, I. Memisevic, R.
    Putting the skeleton back in the closet
    [html] [bibtex]
  • 2019

  • 2019 Materzynska, J., Berger G., Bax, I., Memisevic, R.
    The jester dataset: A large-scale video dataset of human gestures
    ICCV workshop HANDS 2019
    [pdf]
  • 2018

  • 2018 Mahdisoltani, F., Berger, G., Gharbieh, W., Memisevic, R., Fleet, D.
    The more fine-grained, the better for transfer learning
    NeurIPS Workshop on Modeling and Decision-Making in the Spatiotemporal Domain 2018
    [pdf]

  • 2018 Goyal, R., Mahdisoltani, F., Berger, G., Gharbieh, W., Bax, I., Memisevic R.
    Evaluating visual" common sense" using fine-grained classification and captioning tasks
    ICLR workshop
    [pdf]

  • 2018 Mahdisoltani, F., Berger, G., Gharbieh, W., Fleet, D., Memisevic R.
    On the effectiveness of task granularity for transfer learning
    arXiv [pdf]
  • 2017

  • 2017 Goyal, R., Kahou, S., Michalski, V. Materzynska, J., Westphal, *, Bax., I., Memisevic, R.
    * see paper for additional authors.
    The "something something" video database for learning and evaluating visual common sense
    International Conference on Computer Vision (ICCV 2017).
    arXiv [pdf]

  • 2017 Ebrahimi Kahou, S., Michalski, V., Memisevic, R., Pal, C., Vincent, P.
    RATM: Recurrent attentive tracking model
    CVPR workshop [pdf]
  • 2016

  • 2016 Berger, G., Memisevic R.
    Incorporating long-range consistency in CNN-based texture generation
    arXiv [pdf] (accepted at ICLR 2017)

  • 2016 Lin, Z., Courbariaux M., Memisevic, R., Bengio Y.
    Neural Networks with Few Multiplications
    International Conference on Learning Representations (ICLR 2016)
    [pdf]

  • 2016 Krueger, D., Memisevic, R.
    Regularizing RNNs by Stabilizing Activations
    International Conference on Learning Representations (ICLR 2016)
    [pdf]

  • 2016 Im, D., Belghazi, M., Memisevic, R.
    Conservativeness of Untied Auto-Encoders
    Thirtieth AAAI Conference on Artificial Intelligence (AAAI 2016)

  • 2016 de Vries, H., Memisevic, R., Courville, A.
    Deep Learning Vector Quantization
    European Symposium on Artificial Neural Networks 2016

  • 2016 Zhang ,S., Wu, Y., Che, T., Lin, Z, Memisevic, R., Salakhutdinov, R., Bengio, Y.
    Architectural Complexity Measures of Recurrent Neural Networks
    Neural Information Processing Systems (NIPS 2016)
  • 2015

  • 2015 Kamyshanska, H., Memisevic, R.
    The potential energy of an autoencoder
    IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)
    preprint

  • 2015 Konda, K., Memisevic, R., Krueger, D.
    Zero-bias autoencoders and the benefits of co-adapting features
    International Conference on Learning Representations (ICLR 2015) [code]

  • 2015 Konda, K., Memisevic, R.
    Learning visual odometry with a convolutional network
    International Conference on Computer Vision Theory and Applications (VISAPP 2015)
    [pdf]

  • 2015 Jean, S., Cho, K., Memisevic, R., Bengio, Y.
    On using very large target vocabulary for neural machine translation
    Annual meeting of the Association for Computational Linguistics (ACL 2015)

  • 2015 Konda, K., Chandrasekariah, P., Memisevic, R., Triesch, J.
    Real-time activity recognition via deep learning of motion features
    European Symposium on Artificial Neural Networks (ESANN2015)

  • 2015 Kahou, S.E., Michalski, V., Konda, K., Memisevic, R., and Pal, C.
    Recurrent Neural Networks for Emotion Recognition in Video
    To appear in the proceedings of the 17th ACM International Conference on Multimodal Interaction (ICMI 2015).

  • 2015 Kahou, S.E., Michalski V., Memisevic R.
    RATM: Recurrent Attentive Tracking Model
    arXiv:1510.08660 [pdf]

  • 2015 Kahou, S. E., Bouthillier, X., *, P., Courville, A., Vincent, P., Memisevic, R., Pal, C., Bengio, Y.
    EmoNets: Multimodal deep learning approaches for emotion recognition in video
    * see paper for additional authors.
    Journal on Multimodal User Interfaces
    [pdf]
  • 2014

  • 2014 Michalski, V., Memisevic, R., Konda, K.
    Modeling Deep Temporal Dependencies with Recurrent "Grammar Cells"
    Neural Information Processing Systems (NIPS 2014) [pdf]
    an early version of this paper available on arXiv:1402.2333 [pdf]

  • 2014 Konda, K., Memisevic, R., Michalski, V.
    Learning to encode motion using spatio-temporal synchrony
    International Conference on Learning Representations (ICLR 2014)
    [pdf]

  • 2014 Memisevic, R., Konda, K., Krueger, D.
    Zero-bias autoencoders and the benefits of co-adapting features
    arXiv:1402.3337
    [pdf]

  • 2014 Konda, K., Memisevic, R.
    A unified approach to learning depth and motion features (ICVGIP 2014)
  • 2013

  • 2013 Konda, K., Memisevic, R.
    Unsupervised learning of depth and motion.
    arXiv:1312.3429v2
    [pdf]

  • 2013 Kahou, S. E., Pal, C., Bouthillier, X., *, Memisevic, R., Vincent, P., Courville, A. and Bengio, Y.
    Combining Modality Specific Deep Neural Networks for Emotion Recognition in Video
    ICMI 2013
    * see paper for additional authors.
    The paper describes our model that won the ICMI 2013 Grand Challenge on Emotion Recognition in the Wild.
    [pdf]

  • 2013 Kamyshanska, H., Memisevic, R.
    On autoencoder scoring.
    International Conference on Machine Learning (ICML 2013) (oral)
    [pdf] [bibtex]

  • 2013 Memisevic, R., Exarchakis, G.
    Learning invariant features by harnessing the aperture problem.
    International Conference on Machine Learning (ICML 2013)
    [pdf] [bibtex]

  • 2013 Memisevic, R.
    Learning to relate images.
    IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)
    Special issue on Learning Deep Architectures
    [preprint pdf] © IEEE [bibtex]

  • 2013 Bauer, F., Memisevic, R.
    Feature grouping from spatially constrained multiplicative interaction.
    International Conference on Learning Representations (ICLR 2013) (oral)
    [pdf]

  • 2012

  • 2012 Memisevic, R.
    On multi-view feature learning.
    International Conference on Machine Learning (ICML 2012) (oral)
    [pdf] [bibtex] [code]

  • 2012 Memisevic, R., Sigal, L., Fleet, D.
    Shared Kernel Information Embedding for Discriminative Inference.
    IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)
    [pdf] © IEEE [code] [bibtex]

  • 2011

  • 2011 Memisevic, R.
    Gradient-based learning of higher-order image features.
    International Conference on Computer Vision (ICCV 2011).
    [pdf] [bibtex] © IEEE [Website]

  • 2011 Susskind, J., Memisevic, R., Hinton, G., Pollefeys, M.
    Modeling the joint density of two images under a variety of transformations.
    Computer Vision and Pattern Recognition (CVPR 2011).
    [pdf]
    [bibtex] [code] © IEEE

  • 2011 Memisevic, R.
    Learning to relate images: Mapping units, complex cells and simultaneous eigenspaces.
    arXiv:1110.0107v1 [pdf] [bibtex]

  • 2011 Memisevic, R., Conrad, C.
    Depth as a latent variable.
    Frontiers in Neuroscience Conference Abstract: BC11
    [link]

  • 2011 Memisevic, R.
    On spatio-temporal sparse coding: Analysis and an algorithm.
    NIPS workshop on Deep Learning and Unsupervised Feature Learning 2011
    [pdf]

  • 2011 Memisevic, R., Conrad, C.
    Stereopsis via Deep Learning.
    NIPS workshop on Deep Learning and Unsupervised Feature Learning 2011
    [pdf]

  • 2010

  • 2010 Memisevic, R., Hinton, G.
    Learning to Represent Spatial Transformations with Factored Higher-Order Boltzmann Machines.
    June 2010 edition, Vol. 22, No. 6: 1473-1492, Journal Neural Computation.
    [pdf], [bibtex], [Website]
    related 2009 technical report: [pdf]

  • 2010 Memisevic, R., Zach, C., Hinton, G., Pollefeys, M.
    Gated Softmax Classification.
    Neural Information Processing Systems (NIPS 2010).
    [pdf], [bibtex], [Website].
  • 2009 and before

  • 2009 Sigal, L., Memisevic, R., Fleet, D.
    Shared kernel information embedding for discriminative inference.
    Computer Vision and Pattern Recognition (CVPR 2009 © IEEE).
    [pdf]

  • 2008 Memisevic, R.
    Non-linear latent factor models for revealing structure in high-dimensional data.
    PhD thesis, University of Toronto 2008.
    [pdf], [bibtex]

  • 2007 Memisevic, R. and Hinton, G. E.
    Unsupervised learning of image transformations.
    Computer Vision and Pattern Recognition (CVPR 2007 © IEEE).
    [pdf], [bibtex]
    (Related Technical Report: [pdf])

  • 2007 Samulowitz, H. and Memisevic, R.
    Learning to solve QBF.
    Twenty-Second Conference on Artificial Intelligence (AAAI 2007).
    [pdf][bibtex]

  • 2006 Memisevic, R.
    An introduction to structured discriminative learning. Technical report.
    University of Toronto, 2006. [ps],[pdf]

  • 2006 Memisevic, R.
    Dual optimization of conditional probability models.
    NIPS Workshop on Kernel Methods and Structured Domains
    (Technical report, University of Toronto, 2006: [ps],[pdf])

  • 2006 Memisevic, R.
    Kernel Information Embeddings.
    23rd International Conference on Machine Learning (ICML 2006).
    [pdf],[bibtex], [more info], [python code], [GPU version].

  • 2006 Memisevic, R.
    Propagating Errors and Beliefs for Large Scale Nonlinear Structure Prediction.
    North East Student Colloquium on Artificial Intelligence. Ithaca, NY.
    [pdf],[bibtex]

  • 2005 Meinicke, P., Klanke, S., Memisevic, R., Ritter, H.
    Principal Surfaces from Unsupervised Kernel Regression.
    IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)
    Vol. 27, no. 9, pp. 1379-1391. [bibtex]

  • 2005 Memisevic, R. and Hinton, G. E.
    Improving dimensionality reduction with spectral gradient descent.
    Journal Neural Networks.
    18, pp 702-710.
    [online version],[bibtex]

  • 2005 Memisevic, R. and Hinton, G. E.
    Embedding via clustering: Using spectral information to guide dimensionality reduction.
    International Joint Conference on Neural Networks 2005.

  • 2004 Memisevic, R. and Hinton, G. E.
    Multiple Relational Embedding.
    Advances in Neural Information Processing Systems (NIPS 2004).
    [ps.gz],[pdf],[bibtex]

  • 2003 Memisevic, R.
    Unsupervised Kernel Regression for Nonlinear Dimensionality Reduction.
    Master's Thesis, Bielefeld University, 2003.
    [pdf]

Professional activities

  • Senior PC, PC or reviewer for CVPR 2004, 2005, 2010, 2011 (PC), 2012 (PC); NIPS 2006, 2007, 2008, 2009, 2011; ICCV 2010, 2011 (PC); ECCV 2010; DAGM 2011 (Senior PC), 2012 (Senior PC); ICML 2006, 2008 (PC), 2012 (PC).
  • Reviewer for IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Journal of Machine Learning Research (JMLR), Journal Image and Vision Computing (IMAVIS), Journal Constraint, Journal Machine Learning, IEEE Transactions on Image Processing, International Journal of Adaptive Control and Signal Processing.
  • Grant reviewer, Swiss National Science Foundation.
  • Invited speaker at the CIFAR ncap summer school 2013, Toronto.
  • Invited speaker at the IPAM summer school on deep learning, UCLA.
  • I gave tutorials at CVPR 2012 and DAGM 2011 on higher-order feature learning and on spatio-temporal learning [CVPR tutorial website].

Teaching

Miscellaneous


Contact information

Roland Memisevic
Universite de Montreal
Pavillon Andre-Aisenstadt, Departement d'Informatique et de Recherche Operationelle
CP 6128 Succursale Centre-Ville
Montreal QC H3C 3J7
CANADA
email: memisevr[at]iro[dot]umontreal[dot]ca