Jeffrey Mark Siskind's Papers

Refereed Journal Publications

  • J.M. Siskind, `Grounding Language in Perception,' Artificial Intelligence Review, 8:371-391, 1995.

  • J.M. Siskind, `A Computational Study of Cross-Situational Techniques for Learning Word-to-Meaning Mappings,' Cognition, 61(1-2):39-91, October/November 1996. Also appeared in Computational Approaches to Language Acquisition, M.R. Brent, ed., Elsevier, pp. 39-91, 1996.

  • R. Mann, A.D. Jepson, and J.M. Siskind, `The Computational Perception of Scene Dynamics,' Computer Vision and Image Understanding (CVIU), 65(2):113-128, February 1997.

  • J.M. Siskind, `Grounding the Lexical Semantics of Verbs in Visual Perception Using Force Dynamics and Event Logic,' Journal of Artificial Intelligence Research (JAIR), 15:31-90, August 2001. Also available as Technical Report 2000-105, NEC Research Institute, Inc., July 2000.

  • M.R. Brent and J.M. Siskind, `The Role of Exposure to Isolated Words in Early Vocabulary Development,' Cognition, 81/82:B33-B44, September 2001. Also available as Technical Report 99-107, NEC Research Institute, Inc., July 1999. Revised as Technical Report 2000-067R, NEC Research Institute, Inc., May 2000.

  • A.P. Fern, R.L. Givan, and J.M. Siskind, `Specific-to-General Learning for Temporal Events with Application to Learning Event Definitions from Video,' Journal of Artificial Intelligence Research (JAIR), 17:379-449, December 2002.

  • S. Wang and J.M. Siskind, `Image Segmentation with Ratio Cut,' IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 25(6):675-690, June 2003.

  • J.M. Siskind, `Reconstructing Force-Dynamic Models from Video Sequences,' Artificial Intelligence (AIJ), 151(1-2):91-154, December 2003.

  • S. Wang, T. Kubota, J.M. Siskind, and J. Wang, `Salient Closed Boundary Extraction with Ratio Contour,' IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 27(4):546-561, April 2005.

  • W. Wang, I. Pollak, T.-S. Wong, C.A. Bouman, M.P. Harper, and J.M. Siskind, `Hierarchal Stochastic Image Grammars for Classification and Segmentation,' IEEE Transactions on Image Processing (TIP), 15(10):3033-3052, October 2006.

  • J.M. Siskind, J. Sherman, Jr., I. Pollak, M.P. Harper, and C.A. Bouman, `Spatial Random Tree Grammars for Modeling Hierarchal Structure in Images with Regions of Arbitrary Shape,' IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 29(9):1504-1519, September 2007.

  • B.A. Pearlmutter and J.M. Siskind, `Reverse-Mode AD in a Functional Framework: Lambda the Ultimate Backpropagator,' ACM Transactions on Programming Languages and Systems (TOPLAS), 30(2):1-36, April 2008.

  • J.M. Siskind and B.A. Pearlmutter, `Nesting Forward-Mode AD in a Functional Framework,' Higher-Order and Symbolic Computation (HOSC), 21(4):361-376, December 2008.

  • N. Siddharth, A. Barbu, and J.M. Siskind, `Seeing Unseeability to See the Unseeable,' Advances in Cognitive Systems (ACS), 2:77-94, December 2012.

  • A. Barbu, N. Siddharth, A. Michaux, and J.M. Siskind, `Simultaneous Object Detection, Tracking, and Event Recognition,' Advances in Cognitive Systems (ACS), 2:203-220, December 2012.

  • H. Yu, N. Siddharth, A. Barbu, and J.M. Siskind, `A Compositional Framework for Grounding Language Inference, Generation, and Acquisition in Video,' Journal of Artificial Intelligence Research (JAIR), 52:601-713, April 2015.

  • O. Manzyuk, B.A. Pearlmutter, A.A. Radul, D.R. Rush, and J.M. Siskind, `Confusion of Tagged Perturbations in Forward Automatic Differentiation of Higher-Order Functions,' Higher-Order and Symbolic Computation (HOSC), accepted.

    Refereed Conference Publications

  • J.M. Siskind, J.R. Southard, and K.W. Crouch, `Generating Custom High Performance VLSI Designs from Succinct Algorithmic Descriptions,' Proceedings of the Conference on Advanced Research in VLSI, pp. 28-40, January 1982.

  • J.M. Siskind, `Acquiring Core Meanings of Words, Represented as Jackendoff-Style Conceptual Structures, From Correlated Streams of Linguistic and Non-Linguistic Input,' Proceedings of the Twenty Eighth Annual Meeting of the Association for Computational Linguistics (ACL), pp. 143-156, June 1990.

  • J.M. Siskind and D.A. McAllester, `Nondeterministic Lisp as a Substrate for Constraint Logic Programming,' Proceedings of the Eleventh National Conference on Artificial Intelligence (AAAI), pp. 133-138, July 1993.

  • J.M. Siskind, `Lexical Acquisition in the Presence of Noise and Homonymy,' Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI) , pp. 760-766, July 1994.

  • J.M. Siskind and Q. Morris, `A Maximum-Likelihood Approach to Visual Event Classification,' Proceedings of the Fourth European Conference on Computer Vision (ECCV) , pp. 347-360, April 1996.

  • R. Mann, A.D. Jepson, and J.M. Siskind, `The Computational Perception of Scene Dynamics,' Proceedings of the Fourth European Conference on Computer Vision (ECCV) , pp. 528-539, April 1996.

  • J.M. Siskind, `Visual Event Classification via Force Dynamics,' Proceedings of the Seventeenth National Conference on Artificial Intelligence (AAAI), pp. 149-155, August 2000. Also available as Technical Report 2000-007, NEC Research Institute, Inc., January 2000. Revised as Technical Report 2000-047R, NEC Research Institute, Inc., April 2000.

  • S. Wang and J.M. Siskind, `Image Segmentation with Minimum Mean Cut,' Proceedings of the Eighth International Conference on Computer Vision (ICCV), pp. 517-524, July 2001. Also available as Technical Report 2000-169, NEC Research Institute, Inc., December 2000.

  • A.P. Fern, R.L. Givan, and J.M. Siskind, `Specific-to-General Learning for Temporal Events,' Proceedings of the Eighteenth National Conference on Artificial Intelligence (AAAI), pp. 152-158, July 2002.

  • A.P. Fern, J.M. Siskind, and R.L. Givan, `Learning Temporal, Relational, Force-Dynamic Event Definitions from Video,' Proceedings of the Eighteenth National Conference on Artificial Intelligence (AAAI), pp. 159-166, July 2002.

  • I. Pollak, J.M. Siskind, M.P. Harper, and C.A. Bouman, `Modeling and Estimation of Spatial Random Trees with Application to Image Classification,' Proceedings of the 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), volume III, pp. 305-308, Hong Kong 6-12 April 2003.

  • I. Pollak, J.M. Siskind, M.P. Harper, and C.A. Bouman, `Parameter Estimation for Spatial Random Trees Using the EM Algorithm,' Proceedings of the International Conference on Image Processing (ICIP) , pp. 257-260, Barcelona, Spain September 2003.

  • S. Wang, T. Kubota, and J.M. Siskind, `Salient Boundary Detection Using Ratio Contour,' Proceedings of the Neural Information Processing Systems Conference (NIPS), pp. 1571-1578, Vancouver, Canada, 9-11 December 2003.

  • B.A. Pearlmutter and J.M. Siskind, `Lazy Multivariate Higher-Order Forward-Mode AD,' Proceedings of the 34th Annual Symposium on Principles of Programming Languages (POPL), pp. 155-160, Nice, France, 17-19 January 2007.

  • J.M. Siskind and B.A. Pearlmutter, `First-Class Nonstandard Interpretations by Opening Closures,' Proceedings of the 34th Annual Symposium on Principles of Programming Languages (POPL), pp. 71-76, Nice, France, 17-19 January 2007.

  • B.A. Pearlmutter and J.M. Siskind, `Using Programming Language Theory to Make AD Sound and Efficient,' Proceedings of the 5th International Conference on Automatic Differentiation (AD2008), pp. 79-90, Bonn, Germany, 11-15 August 2008.

  • A. Barbu, N. Siddharth, and J.M. Siskind, `Learning Physically-Instantiated Game Play Through Visual Observation,' Proceedings of the 2010 IEEE International Conference on Robotics and Automation (ICRA), pp. 1879-1886, Anchorage, AK, 3-8 May 2010.

  • N. Siddharth, A. Barbu, and J.M. Siskind, `A Visual Language Model for Estimating Object Pose and Structure in a Generative Visual Domain,' Proceedings of the 2011 IEEE International Conference on Robotics and Automation (ICRA), pp. 4854-4860, Shanghai, China, 9-13 May 2011.

  • D. Wingate, N. Goodman, A. Stuhlmueller, and J.M. Siskind, `Nonstandard Interpretations of Probabilistic Programs for Efficient Inference,' Proceedings of the Neural Information Processing Systems Conference (NIPS), Granada, Spain, 12-15 December 2011.

  • Z. Zhang, S. Fidler, J.W. Waggoner, Y. Cao, S. Dickinson, J.M. Siskind, and S. Wang, `Superedge Grouping for Object Localization by Combining Appearance and Shape Information,' Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3266-3273, Providence, RI, 16-21 June 2012.

  • A. Radul, B.A. Pearlmutter, and J.M. Siskind, `AD in Fortran: Implementation via Prepreprocessor,' Proceedings of the 6th International Conference on Automatic Differentiation (AD2012), pp. 273-284, Fort Collins, CO, 23-27 July 2012.

  • A. Barbu, A. Bridge, Z. Burchill, D. Coroian, S. Dickinson, S. Fidler, A. Michaux, S. Mussman, N. Siddharth, D. Salvi, L. Schmidt, J. Shangguan, J.M. Siskind, J. Waggoner, S. Wang, J. Wei, Y. Yin, and Z. Zhang, `Video in Sentences out,' Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 102-112, Catalina, CA, 15-17 August 2012.

  • N. Siddharth, A. Barbu, and J.M. Siskind, `Seeing Unseeability to See the Unseeable,' First Annual Conference for Advances in Cognitive Systems (ACS), pp. 77-94, Palo Alto, CA, 6-8 December 2012.

  • A. Barbu, N. Siddharth, A. Michaux, and J.M. Siskind, `Simultaneous Object Detection, Tracking, and Event Recognition,' First Annual Conference for Advances in Cognitive Systems (ACS), pp. 203-220, Palo Alto, CA, 6-8 December 2012.

  • Y. Cao, D. Barrett, A. Barbu, N. Siddharth, H. Yu, A. Michaux, Y. Lin, S. Dickinson, J.M. Siskind, and S. Wang, `Recognize Human Activities from Partially Observed Videos,' Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2658-2665, Portland, OR, 25-27 June 2013.

  • H. Yu and J.M. Siskind, `Grounded Language Learning from Video Described with Sentences,' Proceedings of the Fifty First Annual Meeting of the Association for Computational Linguistics (ACL), pp. 56-63, Sofia, Bulgaria, 4-9 August 2013.

  • N. Siddharth, A. Barbu, and J.M. Siskind, `Seeing What You're Told: Sentence-Guided Activity Recognition In Video,' Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 732-739, Columbus, OH, 24-27 June 2014.

  • A. Barbu, D.P. Barrett, W. Chen, N. Siddharth, C. Xiong, J.J. Corso, C.D. Fellbaum, C. Hanson, S.J. Hanson, S. Helie, E. Malaia, B.A. Pearlmutter, J.M. Siskind, T.M. Talavage, and R.B. Wilbur, `Seeing is Worse than Believing: Reading People's Minds Better than Computer-Vision Methods Recognize Actions,' Proceedings of the Thirteenth European Conference on Computer Vision (ECCV), pp. 612-627, Zurich, Switzerland, 8-11 September 2014.

  • J.M. Siskind `Conducting Neuroscience to Guide the Development of AI,' Proceedings of the Twenty-Ninth Conference on Artificial Intelligence (AAAI), pp. 4067-4072, Austin, TX, 25-30 January 2015.

  • H. Yu and J.M. Siskind `Learning to Describe Video with Weak Supervision by Exploiting Negative Sentential Information,' Proceedings of the Twenty-Ninth Conference on Artificial Intelligence (AAAI), pp. 3855-3863, Austin, TX, 25-30 January 2015.

    Patents Awarded

  • J.M. Siskind, `A Stability-Checking Algorithm for Two-Dimensional Polygonal Scenes,' US patent 6,693,630, 17 February 2004.

  • J.M. Siskind, `Method for Computing all Occurrences of a Compound Event from Occurrences of Primitive Events,' US patent 6,941,290, 6 September 2005.

  • J.M. Siskind and B.A. Pearlmutter, `Map-Closure: A General Purpose Mechanism for Nonstandard Interpretation,' US patent 8,281,299, 2 October 2012.

  • J.M. Siskind and B.A. Pearlmutter, `Automatic Derivative Method for a Computer Programming Language,' US patent 8,739,137, 27 May 2014.

    Patent Applications Filed

  • J.M. Siskind, A. Barbu, N. Siddharth, and H. Yu, `Correlating Videos and Sentences,' US patent application 14/099,300, filed 6 December 2013.

    Book Chapters

  • J.M. Siskind, `Visual Event Perception,' in Symbolic Visual Learning, K. Ikeuchi and M. Veloso ed., Oxford University Press, chapter 9, pp. 225-263, January 1997.

  • J.M. Siskind, `Learning Word-to-Meaning Mappings,' in Models of Language Acquisition: Inductive and Deductive Approaches, P. Broeder and J. Murre ed., Oxford University Press, chapter 7, pp. 121-153, July 2000.

    Workshop Publications

  • J.M. Siskind, `Dispelling Myths about Language Bootstrapping,' Proceedings of the AAAI Spring Symposium Workshop on Machine Learning of Natural Language and Ontology, pp. 157-164, March 1991.

  • J.M. Siskind, `Naive Physics, Event Perception, Lexical Semantics and Language Acquisition,' Proceedings of the AAAI Spring Symposium Workshop on Machine Learning of Natural Language and Ontology, pp. 165-168, March 1991.

  • J.M. Siskind, `Solving a Lexical Acquisition Task via an Encoding as a Propositional Satisfiability Problem,' Proceedings of the CUNY Sentence Processing Conference, p. 82, March 1993.

  • J.M. Siskind, `Grounding Language in Perception,' Proceedings of the Annual Conference of the Society of Photo-Optical Instrumentation Engineers (SPIE), volume 2055, Intelligent Robots and Computer Vision XII, pp. 198-211, September 1993.

  • J.M. Siskind, `Axiomatic Support for Event Perception,' Proceedings of the AAAI Workshop on Integration of Natural Language and Vision Processing, pp. 153-160, August 1994.

  • J.M. Siskind, `Robust Lexical Acquisition Despite Extremely Noisy Input,' Proceedings of the 19th Boston University Conference on Language Development, D. MacLaughlin and S. McEwen ed., Cascadilla Press, pp. 587-598, March 1995.

  • J.M. Siskind, `Unsupervised Learning of Visually-Observed Events,' Proceedings of the AAAI Fall Symposium Workshop on Learning Complex Behaviors in Adaptive Intelligent Systems, pp. 82-83, November 1996.

  • J.M. Siskind, `Visual Event Perception,' Proceedings of the Ninth NEC Research Symposium, August 1998. Also available as Technical Report 99-033, NEC Research Institute, Inc., March 1999.

  • I. Pollak, J.M. Siskind, M.P. Harper, and C.A. Bouman, `Multiscale Random Tree Models,' IS&T/SPIE 15th Annual Symposium on Electronic Imaging, Santa Clara, CA, 21-24 January 2003.

  • J.M. Siskind, I. Pollak, M.P. Harper, and C.A. Bouman, `Stochastic Grammars for Images on Arbitrary Graphs,' Proceedings of the IEEE Workshop on Statistical Signal Processing, St. Louis, MO, 28 September-1 October 2003.

  • I. Pollak, J.M. Siskind, M.P. Harper, and C.A. Bouman, `Modeling of Images with Spatial Random Trees,' SIAM Conference on Imaging Science, Salt Lake City, UT, 3-5 May 2004.

  • J.M. Siskind and B.A. Pearlmutter, `Perturbation Confusion and Referential Transparency: Correct Functional Implementation of Forward-Mode AD,' Draft Proceedings of the 17th International Workshop on Implementation and Application of Functional Languages (IFL2005), Dublin, Ireland, 19-21 September 2005.

  • A. Barbu, N. Siddharth, and J.M. Siskind, `Language-Driven Video Retrieval,' Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshop on Vision Meets Cognition (CVPR Workshop), Columbus, OH, 23 June 2014.

    Technical Reports

  • J.M. Siskind, `The Culprit Pointer Method for Selective Backtracking,' S.M. thesis, Artificial Intelligence Laboratory, MIT, January 1989.

  • D.A. McAllester and J.M. Siskind, `Lifting Transformations,' AI Memo 1343, Artificial Intelligence Laboratory, MIT, December 1991.

  • J.M. Siskind, `Naive Physics, Event Perception, Lexical Semantics and Language Acquisition,' Ph.D. thesis, Artificial Intelligence Laboratory, MIT, January 1992.

  • J.M. Siskind and D.A. McAllester, `Screamer: A Portable Efficient Implementation of Nondeterministic Common Lisp,' Technical Report IRCS-93-03, Institute for Research in Cognitive Science, University of Pennsylvania, 1993.

  • J.M. Siskind, `Lexical Acquisition as Constraint Satisfaction,' Technical Report IRCS-93-41, Institute for Research in Cognitive Science, University of Pennsylvania, 1993.

  • J.M. Siskind, `Flow-Directed Lightweight Closure Conversion,' Technical Report 99-105, NEC Research Institute, Inc., July 1999. Revised as Technical Report 99-190R, NEC Research Institute, Inc., December 1999.

  • S. Wang and J.M. Siskind, `Image Segmentation with Ratio Cut---Extended Version,' Technical Report TR-ECE-02-07, School of Electrical and Computer Engineering, Purdue University, 2002.

  • I. Pollak, J.M. Siskind, M.P. Harper, and C.A. Bouman, `Spatial Random Trees and the Center-Surround Algorithm,' Technical Report TR-ECE-03-03, School of Electrical and Computer Engineering, Purdue University, 2003.

  • J.M. Siskind and B.A. Pearlmutter, `Using Polyvariant Union-Free Flow Analysis to Compile a Higher-Order Functional-Programming Language with a First-Class Derivative Operator to Efficient Fortran-like Code,' Technical Report TR-ECE-08-01, School of Electrical and Computer Engineering, Purdue University, 2008.

  • J.M. Siskind and B.A. Pearlmutter, `Putting the Automatic Back into AD: Part I, What's Wrong,' Technical Report TR-ECE-08-02, School of Electrical and Computer Engineering, Purdue University, 2008.

  • B.A. Pearlmutter and J.M. Siskind, `Putting the Automatic Back into AD: Part II, Dynamic, Automatic, Nestable, and Fast,' Technical Report TR-ECE-08-03, School of Electrical and Computer Engineering, Purdue University, 2008.

  • J.M. Siskind and B.A. Pearlmutter, `Nesting Forward-Mode AD in a Functional Framework,' Technical Report TR-ECE-08-09, School of Electrical and Computer Engineering, Purdue University, 2008.

  • J.M. Siskind, `Grounding the Lexical Semantics of Verbs in Visual Perception Using Force Dynamics and Event Logic,' arXiv:1106.0256, 1 June 2011.

  • A.P. Fern, R.L. Givan, and J.M. Siskind, `Specific-to-General Learning for Temporal Events with Application to Learning Event Definitions from Video,' arXiv:1106.4572, 22 June 2011.

  • A. Radul, B.A. Pearlmutter, and J.M. Siskind, `AD in Fortran, Part 1: Design,' arXiv:1203.1448. 7 March 2012.

  • A. Radul, B.A. Pearlmutter, and J.M. Siskind, `AD in Fortran, Part 2: Implementation via Prepreprocessor,' arXiv:1203.1450, 7 March 2012.

  • A. Barbu, A. Michaux, N. Siddharth, and J.M. Siskind, `Simultaneous Object Detection, Tracking, and Event Recognition,' arXiv:1204.2741, 12 April 2012

  • A. Barbu, A. Bridge, Z. Burchill, D. Coroian, S. Dickinson, S. Fidler, A. Michaux, S. Mussman, N. Siddharth, D. Salvi, L. Schmidt, J. Shangguan, J.M. Siskind, J. Waggoner, S. Wang, J. Wei, Y. Yin, and Z. Zhang, `Video In Sentences Out,' arXiv:1204.2742, 12 April 2012.

  • N. Siddharth, A. Barbu, and J.M. Siskind, `Seeing Unseeability to See the Unseeable,' arXiv:1204.2801, 12 April 2012.

  • A. Barbu, A. Bridge, D. Coroian, S. Dickinson, S. Mussman, N. Siddharth, D. Salvi, L. Schmidt, J. Shangguan, J.M. Siskind, J. Waggoner, S. Wang, J. Wei, Y. Yin, and Z. Zhang, `Large-Scale Automatic Labeling of Video Events with Verbs Based on Event-Participant Interaction,' arXiv:1204.3616, 16 April 2012.

  • O. Manzyuk, B.A. Pearlmutter, A.A. Radul, D.R. Rush, and J.M. Siskind, `Confusion of Tagged Perturbations in Forward Automatic Differentiation of Higher-Order Functions,' arXiv:1211.4892, 20 November 2012.

  • A. Barbu, N. Siddharth, C. Xiong, J.J. Corso, C.D. Fellbaum, C. Hanson, S.J. Hanson, S. Helie, E. Malaia, B.A. Pearlmutter, J.M. Siskind, T.M. Talavage, and R.B. Wilbur, `The Compositional Nature of Verb and Argument Representations in the Human Brain,' arxiv:1306.2293, 10 June 2013.

  • D.P. Barrett and J.M. Siskind, `Felzenszwalb-Baum-Welch: Event Detection by Changing Appearance,' arXiv:1306.4746, 20 June 2013.

  • H. Yu and J.M. Siskind, `Discriminative Training: Learning to Describe Video with Sentences, from Video Described with Sentences,' arXiv:1306.5263, 21 June 2013.

  • N. Siddharth, A. Barbu, and J.M. Siskind, `Seeing What You're Told: Sentence-Guided Activity Recognition in Video,' arxiv:1308.4189, 19 August 2013.

  • A. Barbu, N. Siddharth, and J.M. Siskind, `Saying What You're Looking For: Linguistics Meets Video Search,' arxiv:1309.5174, 20 September 2013.

  • A. Barbu, A. Bridge, Z. Burchill, D. Coroian, S. Dickinson, S. Fidler, A. Michaux, S. Mussman, N. Siddharth, D. Salvi, L. Schmidt, J. Shangguan, J.M. Siskind, J. Waggoner, S. Wang, J. Wei, Y. Yin, and Z. Zhang, `Video in sentences out,' arXiv:1408.6418, 9 August 2014.

  • H. Yu, D.P. Barrett, and J.M. Siskind, `A Faster Method for Tracking and Scoring Videos Corresponding to Sentences,' arXiv:1411.4064, 14 November 2014.

  • A. Barbu, N. Siddharth, C. Xiong, J.J. Corso, C.D. Fellbaum, C. Hanson, S.J. Hanson, S. Helie, E. Malaia, B.A. Pearlmutter, J.M. Siskind, T.M. Talavage, and R.B. Wilbur, `The Compositional Nature of Event Representations in the Human Brain,' arXiv:1505.06670, 25 May 2015.

  • H. Yu and J.M. Siskind, `Sentence Directed Video Object Codetection,' arXiv:1506.02059, 5 June 2015.