Posts by Collection

patents_pubs

Narrative Fragment Creation: An Approach for Learning Narrative Knowledge

Published:

We propose the narrative fragment - a sequence of story events - and a method for automatically creating these fragments with narrative generation through partial order planning and analysis through n-gram modeling. The generated plans establish causal and temporal relationships, and by modeling those relationships and creating fragments, our system learns narrative knowledge.

C. Cervantes & W. Fu. (2013) Narrative Fragment Creation: An Approach for Learning Narrative Knowledge. Conference on Advances in Cognitive Systems (ACS) https://cmcervantes.github.io/files/cervantes_2013_narrative.pdf

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

Published:

This paper presents Flickr30k Entities, which augments the 158k captions from Flickr30k with 244k coreference chains linking mentions of the same entities in images, as well as 276k manually annotated bounding boxes corresponding to each entity. We present experiments demonstrating the usefulness of our annotations for text-to-image reference resolution, or the task of localizing textual entity mentions in an image, and for bidirectional image-sentence retrieval.

B. Plummer, L. Wang, C. Cervantes, J. Caicedo, J. Hockenmaier, & S. Lazebnik. (2015) Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. International Conference on Computer Vision (ICCV) https://cmcervantes.github.io/files/plummer_2015_flickr30kEntities.pdf

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

Published:

This journal version of the 2015 paper of the same name adds experiments, analysis, and examples to the existing work.

B. Plummer, L. Wang, C. Cervantes, J. Caicedo, J. Hockenmaier, & S. Lazebnik. (2017) Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. International Journal of Computer Vision (IJCV) https://cmcervantes.github.io/files/plummer_2017_flickr30kEntities.pdf

Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues

Published:

This paper presents a framework for localization or grounding of phrases in images using a large collection of linguistic and visual cues. We model the appearance, size, and position of entity bounding boxes, adjectives that contain attribute information, and spatial relationships between pairs of entities connected by verbs or prepositions. Special attention is given to relationships between people and clothing or body part mentions, as they are useful for distinguishing individuals.

B. Plummer, A. Mallya, C. Cervantes, J. Hockenmaier, & S. Lazebnik. (2017) Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues. International Conference on Computer Vision (ICCV) https://cmcervantes.github.io/files/plummer_2017_phrase.pdf

Entity-Based Scene Understanding

Published:

We define entity-based scene understanding as the task of identifying the entities in a visual scene from multiple descriptions by a) identifying coreference and subset relations between entity mentions, and b) grounding entity mentions to image regions. We apply our models to two datasets (Flickr30K Entities v2 and MSCOCO) and show that grounding can benefit significantly from relation prediction in both cases.

C. Cervantes, B. Plummer, S. Lazebnik, & J. Hockenmaier. (2018) Entity-Based Scene Understanding. Master's Thesis. University of Illinois at Urbana-Champaign https://cmcervantes.github.io/files/cervantes_2018_entity.pdf

Method for Extracting Landmark Graphs from Natural Language Route Instructions

Published:

Landmarks are central to how people navigate, but most navigation technologies do not incorporate them into their representations. We propose the landmark graph generation task (creating landmark-based spatial representations from natural language) and introduce a fully end-to-end neural approach to generate these graphs. We evaluate our models on the SAIL route instruction dataset, as well as on a small set of real-world delivery instructions that we collected, and we show that our approach yields high quality results on both our task and the related robotic navigation task.

C. Cervantes. Method for Extracting Landmark Graphs from Natural Language Route Instructions. U.S. Patent Application 16/774315, filed January 2020. Patent Pending

Method, Apparatus, and System for Providing a Context-Aware Location Representation

Published:

This invention aims to produce dense representations (embeddings) for location entities through the combination of spatial and structured information (e.g. present in a knowledge graph) with unstructured information (e.g. web-scraped text). These representations are constructed using representation learning techniques (e.g. Deepwalk) and traditional natural language processing tools.

S. Kompella & C. Cervantes. Method, Apparatus, and System for Providing a Context-Aware Location Representation. U.S. Patent Application 17/116717, filed December 2020. Patent Pending

Method, Apparatus, and System for Providing a Location Representation for Machine Learning Tasks

Published:

This invention describes a broad mechanism to create real-valued vector representations that encode locations’ semantic and spatial properties for use in downstream tasks like search, question answering, relation prediction, and so on. The invention’s methods draw heavily on prior art, but incorporate significant modifications to capture the complex, high-level multi-model information that defines locations.

C. Cervantes & S. Kompella. Method, Apparatus, and System for Providing a Location Representation for Machine Learning Tasks. U.S. Patent Application 17/116727, filed December 2020. Patent Pending

Method, Apparatus, and System for Providing Semantic Categorization of an Arbitrarily Granular Location

Published:

This invention describes a method for combining locations of arbitrary granularity and predicting semantic categories for those groupings. In this context, a location can refer to anything from an individual place of interest to a larger administrative area like a neighborhood or city. Similarly, a semantic category could be equally expressive, encompassing things like “family friendly,” “safe for travelers,” or “trendy”.

C. Cervantes & S. Kompella. Method, Apparatus, and System for Providing Semantic Categorization of an Arbitrarily Granular Location. U.S. Patent Application 17/116743, filed December 2020. Patent Pending

Method, Apparatus, and System for Combining Location Data Sources

Published:

This invention defines a system to combine related sources of location data (e.g. a knowledge graph), both in the context of finding matching entities and to predict new relationships between existing entities. The proposed system is able to automatically merge and enrich disparate data sources through neural classification models.

C. Cervantes & S. Kompella. Method, Apparatus, and System for Combining Location Data Sources. U.S. Patent Application 17/116756, filed December 2020. Patent Pending

projects

Entity-Based Scene Understanding

Last Meter Delivery

Location Ontology

AI/ML Task Force: Object Detection in 3D Point Clouds

Natural Guidance Features

service_leadership

Graduate Student Ambassador

Served as a resource to help new Ph.D. students acclimate to graduate school.

Graduate Student Advisory Committee

Served as a point of contact between Computer Science graduate students and the department administration, enabling concerns to be voiced and facilitating positive change.

Promoting Undergraduate Research in Engineering

Independently lectured, led discussions, and presented papers to help undergraduates engage in research.

Fellowships, Assistantships, and Admissions Committee

Worked with faculty to review student applications for admission to the Computer Science graduate program and offered a graduate student perspective on the applicants’ merits.

Graduate Study Committee

Worked with faculty to review and adjust academic policies within the Computer Science graduate program.

Graduate Student Diversity Advisory Board

Working with other students and administrative staff, reviewed department initiatives related to equity, diversity, and inclusion in an effort to improve diverse student admission and retention.

talks_teach

Narrative Fragment Creation: An Approach for Learning Narrative

Published:

Presented the paper of the same name at ACS 2013.

Flickr30k Entities

Published:

Presented a poster on the Flickr30k Entities dataset, the annotated image-caption data first introduced in the Flickr30k Entities paper published in ICCV.

Introduction to Natural Language Processing

Created and graded homework assignments, graded exams, and held office hours. The course covered traditional natural language processing topics, including morphology, syntax, parsing, machine translation, generation, etc.

Entity Prediction from Parallel Image Captions

Published:

Presented the then-in-progress work on coreference resolution for parallel image captions, which leveraged both classic linear classification schemes and rule-based anaphora resolution heuristics.

Entity Based Visual Scene Understanding

Published:

Presented work for coreference resolution for parallel image captions, bridging anaphora, and grounding using bidirectional LSTMs, simple feed-forward networks, and a multi-task learning scheme for interrelated vision and language tasks.

Machine Learning for Natural Language Processing

Advised students on independent projects, created and graded homework assignments, graded exams, and held office hours. Topics focused on deep learning techniques for natural language processing.

Last Meter Delivery

Published:

Presented the then-ongoing research and prototypes for the last meter delivery project, including the data collection mobile app and the work-in-progress results from the neural method used to extract spatial representations from natural language route instructions.

Graph Neural Network Working Group

Published:

Presented to the HERE research organization on the theory and practice of graph neural network construction, with particular focus on implementing graph attention networks in Pytorch.