Publications | Max Planck Institute for Intelligent Systems

66 results (View BibTeX file of all listed publications)

2015

Exploiting Object Similarity in 3D Reconstruction

Zhou, C., Güney, F., Wang, Y., Geiger, A.

In International Conference on Computer Vision (ICCV), December 2015 (inproceedings)

Abstract

Despite recent progress, reconstructing outdoor scenes in 3D from movable platforms remains a highly difficult endeavor. Challenges include low frame rates, occlusions, large distortions and difficult lighting conditions. In this paper, we leverage the fact that the larger the reconstructed area, the more likely objects of similar type and shape will occur in the scene. This is particularly true for outdoor scenes where buildings and vehicles often suffer from missing texture or reflections, but share similarity in 3D shape. We take advantage of this shape similarity by locating objects using detectors and jointly reconstructing them while learning a volumetric model of their shape. This allows us to reduce noise while completing missing surfaces as objects of similar shape benefit from all observations for the respective category. We evaluate our approach with respect to LIDAR ground truth on a novel challenging suburban dataset and show its advantages over the state-of-the-art.

avg ps

pdf suppmat [BibTex]

2015

avg ps Zhou, C., Güney, F., Wang, Y., Geiger, A. Exploiting Object Similarity in 3D Reconstruction In International Conference on Computer Vision (ICCV), December 2015 (inproceedings)

pdf suppmat [BibTex]

FollowMe: Efficient Online Min-Cost Flow Tracking with Bounded Memory and Computation

Lenz, P., Geiger, A., Urtasun, R.

In International Conference on Computer Vision (ICCV), International Conference on Computer Vision (ICCV), December 2015 (inproceedings)

Abstract

One of the most popular approaches to multi-target tracking is tracking-by-detection. Current min-cost flow algorithms which solve the data association problem optimally have three main drawbacks: they are computationally expensive, they assume that the whole video is given as a batch, and they scale badly in memory and computation with the length of the video sequence. In this paper, we address each of these issues, resulting in a computationally and memory-bounded solution. First, we introduce a dynamic version of the successive shortest-path algorithm which solves the data association problem optimally while reusing computation, resulting in faster inference than standard solvers. Second, we address the optimal solution to the data association problem when dealing with an incoming stream of data (i.e., online setting). Finally, we present our main contribution which is an approximate online solution with bounded memory and computation which is capable of handling videos of arbitrary length while performing tracking in real time. We demonstrate the effectiveness of our algorithms on the KITTI and PETS2009 benchmarks and show state-of-the-art performance, while being significantly faster than existing solvers.

avg ps

pdf suppmat video project [BibTex]

avg ps Lenz, P., Geiger, A., Urtasun, R. FollowMe: Efficient Online Min-Cost Flow Tracking with Bounded Memory and Computation In International Conference on Computer Vision (ICCV), International Conference on Computer Vision (ICCV), December 2015 (inproceedings)

pdf suppmat video project [BibTex]

Scalable Robust Principal Component Analysis using Grassmann Averages

Hauberg, S., Feragen, A., Enficiaud, R., Black, M.

IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), December 2015 (article)

Abstract

In large datasets, manual data verification is impossible, and we must expect the number of outliers to increase with data size. While principal component analysis (PCA) can reduce data size, and scalable solutions exist, it is well-known that outliers can arbitrarily corrupt the results. Unfortunately, state-of-the-art approaches for robust PCA are not scalable. We note that in a zero-mean dataset, each observation spans a one-dimensional subspace, giving a point on the Grassmann manifold. We show that the average subspace corresponds to the leading principal component for Gaussian data. We provide a simple algorithm for computing this Grassmann Average (GA), and show that the subspace estimate is less sensitive to outliers than PCA for general distributions. Because averages can be efficiently computed, we immediately gain scalability. We exploit robust averaging to formulate the Robust Grassmann Average (RGA) as a form of robust PCA. The resulting Trimmed Grassmann Average (TGA) is appropriate for computer vision because it is robust to pixel outliers. The algorithm has linear computational complexity and minimal memory requirements. We demonstrate TGA for background modeling, video restoration, and shadow removal. We show scalability by performing robust PCA on the entire Star Wars IV movie; a task beyond any current method. Source code is available online.

ps sf

preprint pdf from publisher supplemental Project Page [BibTex]

ps sf Hauberg, S., Feragen, A., Enficiaud, R., Black, M. Scalable Robust Principal Component Analysis using Grassmann Averages IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), December 2015 (article)

preprint pdf from publisher supplemental Project Page [BibTex]

Intrinsic Depth: Improving Depth Transfer with Intrinsic Images

Kong, N., Black, M. J.

In IEEE International Conference on Computer Vision (ICCV), pages: 3514-3522, International Conference on Computer Vision (ICCV), December 2015 (inproceedings)

Abstract

We formulate the estimation of dense depth maps from video sequences as a problem of intrinsic image estimation. Our approach synergistically integrates the estimation of multiple intrinsic images including depth, albedo, shading, optical flow, and surface contours. We build upon an example-based framework for depth estimation that uses label transfer from a database of RGB and depth pairs. We combine this with a method that extracts consistent albedo and shading from video. In contrast to raw RGB values, albedo and shading provide a richer, more physical, foundation for depth transfer. Additionally we train a new contour detector to predict surface boundaries from albedo, shading, and pixel values and use this to improve the estimation of depth boundaries. We also integrate sparse structure from motion with our method to improve the metric accuracy of the estimated depth maps. We evaluate our Intrinsic Depth method quantitatively by estimating depth from videos in the NYU RGB-D and SUN3D datasets. We find that combining the estimation of multiple intrinsic images improves depth estimation relative to the baseline method.

pdf suppmat YouTube official video poster Project Page Project Page [BibTex]

ps Kong, N., Black, M. J. Intrinsic Depth: Improving Depth Transfer with Intrinsic Images In IEEE International Conference on Computer Vision (ICCV), pages: 3514-3522, International Conference on Computer Vision (ICCV), December 2015 (inproceedings)

pdf suppmat YouTube official video poster Project Page Project Page [BibTex]

Detailed Full-Body Reconstructions of Moving People from Monocular RGB-D Sequences

Bogo, F., Black, M. J., Loper, M., Romero, J.

In International Conference on Computer Vision (ICCV), pages: 2300-2308, December 2015 (inproceedings)

Abstract

We accurately estimate the 3D geometry and appearance of the human body from a monocular RGB-D sequence of a user moving freely in front of the sensor. Range data in each frame is first brought into alignment with a multi-resolution 3D body model in a coarse-to-fine process. The method then uses geometry and image texture over time to obtain accurate shape, pose, and appearance information despite unconstrained motion, partial views, varying resolution, occlusion, and soft tissue deformation. Our novel body model has variable shape detail, allowing it to capture faces with a high-resolution deformable head model and body shape with lower-resolution. Finally we combine range data from an entire sequence to estimate a high-resolution displacement map that captures fine shape details. We compare our recovered models with high-resolution scans from a professional system and with avatars created by a commercial product. We extract accurate 3D avatars from challenging motion sequences and even capture soft tissue dynamics.

Video pdf Project Page Project Page [BibTex]

ps Bogo, F., Black, M. J., Loper, M., Romero, J. Detailed Full-Body Reconstructions of Moving People from Monocular RGB-D Sequences In International Conference on Computer Vision (ICCV), pages: 2300-2308, December 2015 (inproceedings)

Video pdf Project Page Project Page [BibTex]

3D Object Reconstruction from Hand-Object Interactions

Tzionas, D., Gall, J.

In International Conference on Computer Vision (ICCV), pages: 729-737, International Conference on Computer Vision (ICCV), December 2015 (inproceedings)

Abstract

Recent advances have enabled 3d object reconstruction approaches using a single off-the-shelf RGB-D camera. Although these approaches are successful for a wide range of object classes, they rely on stable and distinctive geometric or texture features. Many objects like mechanical parts, toys, household or decorative articles, however, are textureless and characterized by minimalistic shapes that are simple and symmetric. Existing in-hand scanning systems and 3d reconstruction techniques fail for such symmetric objects in the absence of highly distinctive features. In this work, we show that extracting 3d hand motion for in-hand scanning effectively facilitates the reconstruction of even featureless and highly symmetric objects and we present an approach that fuses the rich additional information of hands into a 3d reconstruction pipeline, significantly contributing to the state-of-the-art of in-hand scanning.

pdf Project's Website Video Spotlight Extended Abstract YouTube DOI Project Page [BibTex]

ps Tzionas, D., Gall, J. 3D Object Reconstruction from Hand-Object Interactions In International Conference on Computer Vision (ICCV), pages: 729-737, International Conference on Computer Vision (ICCV), December 2015 (inproceedings)

pdf Project's Website Video Spotlight Extended Abstract YouTube DOI Project Page [BibTex]

Untethered Magnetic Micromanipulation

Diller, E., Sitti, M.

In Micro-and Nanomanipulation Tools, 13, 10, Wiley-VCH Verlag GmbH & Co. KGaA, November 2015 (inbook)

Abstract

This chapter discusses the methods and state of the art in microscale manipulation in remote environments using untethered microrobotic devices. It focuses on manipulation at the size scale of tens to hundreds of microns, where small size leads to a dominance of microscale physical effects and challenges in fabrication and actuation. To motivate the challenges of operating at this size scale, the chapter includes coverage of the physical forces relevant to microrobot motion and manipulation below the millimeter-size scale. It then introduces the actuation methods commonly used in untethered manipulation schemes, with particular focus on magnetic actuation due to its wide use in the field. The chapter divides these manipulation techniques into two types: contact manipulation, which relies on direct pushing or grasping of objects for motion, and noncontact manipulation, which relies indirectly on induced fluid flow from the microrobot motion to move objects without any direct contact.

DOI Project Page [BibTex]

pi Diller, E., Sitti, M. Untethered Magnetic Micromanipulation In Micro-and Nanomanipulation Tools, 13, 10, Wiley-VCH Verlag GmbH & Co. KGaA, November 2015 (inbook)

DOI Project Page [BibTex]

SMPL: A Skinned Multi-Person Linear Model

Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M. J.

ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1-248:16, ACM, New York, NY, October 2015 (article)

Abstract

We present a learned model of human body shape and pose-dependent shape variation that is more accurate than previous models and is compatible with existing graphics pipelines. Our Skinned Multi-Person Linear model (SMPL) is a skinned vertex-based model that accurately represents a wide variety of body shapes in natural human poses. The parameters of the model are learned from data including the rest pose template, blend weights, pose-dependent blend shapes, identity-dependent blend shapes, and a regressor from vertices to joint locations. Unlike previous models, the pose-dependent blend shapes are a linear function of the elements of the pose rotation matrices. This simple formulation enables training the entire model from a relatively large number of aligned 3D meshes of different people in different poses. We quantitatively evaluate variants of SMPL using linear or dual-quaternion blend skinning and show that both are more accurate than a Blend-SCAPE model trained on the same data. We also extend SMPL to realistically model dynamic soft-tissue deformations. Because it is based on blend skinning, SMPL is compatible with existing rendering engines and we make it available for research purposes.

pdf video code/model errata DOI Project Page Project Page [BibTex]

ps Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M. J. SMPL: A Skinned Multi-Person Linear Model ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1-248:16, ACM, New York, NY, October 2015 (article)

pdf video code/model errata DOI Project Page Project Page [BibTex]

Proceedings of the 37th German Conference on Pattern Recognition

Gall, J., Gehler, P., Leibe, B.

Springer, German Conference on Pattern Recognition, October 2015 (proceedings)

GCPR conference website [BibTex]

ps Gall, J., Gehler, P., Leibe, B. Proceedings of the 37th German Conference on Pattern Recognition Springer, German Conference on Pattern Recognition, October 2015 (proceedings)

GCPR conference website [BibTex]

Towards Probabilistic Volumetric Reconstruction using Ray Potentials

(Best Paper Award)

Ulusoy, A. O., Geiger, A., Black, M. J.

In 3D Vision (3DV), 2015 3rd International Conference on, pages: 10-18, Lyon, October 2015 (inproceedings)

Abstract

This paper presents a novel probabilistic foundation for volumetric 3-d reconstruction. We formulate the problem as inference in a Markov random field, which accurately captures the dependencies between the occupancy and appearance of each voxel, given all input images. Our main contribution is an approximate highly parallelized discrete-continuous inference algorithm to compute the marginal distributions of each voxel's occupancy and appearance. In contrast to the MAP solution, marginals encode the underlying uncertainty and ambiguity in the reconstruction. Moreover, the proposed algorithm allows for a Bayes optimal prediction with respect to a natural reconstruction loss. We compare our method to two state-of-the-art volumetric reconstruction algorithms on three challenging aerial datasets with LIDAR ground truth. Our experiments demonstrate that the proposed algorithm compares favorably in terms of reconstruction accuracy and the ability to expose reconstruction uncertainty.

avg ps

code YouTube pdf suppmat DOI Project Page [BibTex]

avg ps Ulusoy, A. O., Geiger, A., Black, M. J. Towards Probabilistic Volumetric Reconstruction using Ray Potentials In 3D Vision (3DV), 2015 3rd International Conference on, pages: 10-18, Lyon, October 2015 (inproceedings)

code YouTube pdf suppmat DOI Project Page [BibTex]

Compliant wing design for a flapping wing micro air vehicle

Colmenares, D., Kania, R., Zhang, W., Sitti, M.

In Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on, pages: 32-39, September 2015 (inproceedings)

Abstract

In this work, we examine several wing designs for a motor-driven, flapping-wing micro air vehicle capable of liftoff. The full system consists of two wings independently driven by geared pager motors that include a spring in parallel with the output shaft. The linear transmission allows for resonant operation, while control is achieved by direct drive of the wing angle. Wings used in previous work were chosen to be fully rigid for simplicity of modeling and fabrication. However, biological wings are highly flexible and other micro air vehicles have successfully utilized flexible wing structures for specialized tasks. The goal of our study is to determine if wing flexibility can be generally used to increase wing performance. Two approaches to lift improvement using flexible wings are explored, resonance of the wing cantilever structure and dynamic wing twisting. We design and test several wings that are compared using different figures of merit. A twisted design improved lift per power by 73.6% and maximum lift production by 53.2% compared to the original rigid design. Wing twist is then modeled in order to propose optimal wing twist profiles that can maximize either wing efficiency or lift production.

DOI [BibTex]

pi Colmenares, D., Kania, R., Zhang, W., Sitti, M. Compliant wing design for a flapping wing micro air vehicle In Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on, pages: 32-39, September 2015 (inproceedings)

DOI [BibTex]

Methods of forming dry adhesive structures

Sitti, M., Murphy, M., Aksak, B.

September 2015, US Patent 9,120,953 (patent)

Abstract

Methods of forming dry adhesives including a method of making a dry adhesive including applying a liquid polymer to the second end of the stem, molding the liquid polymer on the stem in a mold, wherein the mold includes a recess having a cross-sectional area that is less than a cross-sectional area of the second end of the stem, curing the liquid polymer in the mold to form a tip at the second end of the stem, wherein the tip includes a second layer stem; corresponding to the recess in the mold, and removing the tip from the mold after the liquid polymer cures.

[BibTex]

pi Sitti, M., Murphy, M., Aksak, B. Methods of forming dry adhesive structures September 2015, US Patent 9,120,953 (patent)

[BibTex]

Millimeter-scale magnetic swimmers using elastomeric undulations

Zhang, J., Diller, E.

In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages: 1706-1711, September 2015 (inproceedings)

Abstract

This paper presents a new soft-bodied millimeterscale swimmer actuated by rotating uniform magnetic fields. The proposed swimmer moves through internal undulatory deformations, resulting from a magnetization profile programmed into its body. To understand the motion of the swimmer, a mathematical model is developed to describe the general relationship between the deflection of a flexible strip and its magnetization profile. As a special case, the situation of the swimmer on the water surface is analyzed and predictions made by the model are experimentally verified. Experimental results show the controllability of the proposed swimmer under a computer vision-based closed-loop controller. The swimmers have nominal dimensions of 1.5×4.9×0.06 mm and a top speed of 50 mm/s (10 body lengths per second). Waypoint following and multiagent control are demonstrated for swimmers constrained at the air-water interface and underwater swimming is also shown, suggesting the promising potential of this type of swimmer in biomedical and microfluidic applications.

link (url) DOI [BibTex]

pi Zhang, J., Diller, E. Millimeter-scale magnetic swimmers using elastomeric undulations In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages: 1706-1711, September 2015 (inproceedings)

link (url) DOI [BibTex]

Perception of Strength and Power of Realistic Male Characters

Wellerdiek, A. C., Breidt, M., Geuss, M. N., Streuber, S., Kloos, U., Black, M. J., Mohler, B. J.

In Proc. ACM SIGGRAPH Symposium on Applied Perception, SAP’15, pages: 7-14, ACM, New York, NY, September 2015 (inproceedings)

Abstract

We investigated the influence of body shape and pose on the perception of physical strength and social power for male virtual characters. In the first experiment, participants judged the physical strength of varying body shapes, derived from a statistical 3D body model. Based on these ratings, we determined three body shapes (weak, average, and strong) and animated them with a set of power poses for the second experiment. Participants rated how strong or powerful they perceived virtual characters of varying body shapes that were displayed in different poses. Our results show that perception of physical strength was mainly driven by the shape of the body. However, the social attribute of power was influenced by an interaction between pose and shape. Specifically, the effect of pose on power ratings was greater for weak body shapes. These results demonstrate that a character with a weak shape can be perceived as more powerful when in a high-power pose.

PDF DOI Project Page [BibTex]

ps Wellerdiek, A. C., Breidt, M., Geuss, M. N., Streuber, S., Kloos, U., Black, M. J., Mohler, B. J. Perception of Strength and Power of Realistic Male Characters In Proc. ACM SIGGRAPH Symposium on Applied Perception, SAP’15, pages: 7-14, ACM, New York, NY, September 2015 (inproceedings)

PDF DOI Project Page [BibTex]

Experimental investigation of optimal adhesion of mushroomlike elastomer microfibrillar adhesives

Marvi, H., Song, S., Sitti, M.

Langmuir, 31(37):10119-10124, American Chemical Society, August 2015 (article)

Abstract

Optimal fiber designs for the maximal pull-off force have been indispensable for increasing the attachment performance of recently introduced gecko-inspired reversible micro/nanofibrillar adhesives. There are several theoretical studies on such optimal designs; however, due to the lack of three-dimensional (3D) fabrication techniques that can fabricate such optimal designs in 3D, there have not been many experimental investigations on this challenge. In this study, we benefitted from recent advances in two-photon lithography techniques to fabricate mushroomlike polyurethane elastomer fibers with different aspect ratios of tip to stalk diameter (β) and tip wedge angles (θ) to investigate the effect of these two parameters on the pull-off force. We found similar trends to those predicted theoretically. We found that β has an impact on the slope of the force-displacement curve while both β and θ play a role in the stress distribution and crack propagation. We found that these effects are coupled and the optimal set of parameters also depends on the fiber material. This is the first experimental verification of such optimal designs proposed for mushroomlike microfibers. This experimental approach could be used to evaluate a wide range of complex microstructured adhesive designs suggested in the literature and optimize them.

DOI [BibTex]

pi Marvi, H., Song, S., Sitti, M. Experimental investigation of optimal adhesion of mushroomlike elastomer microfibrillar adhesives Langmuir, 31(37):10119-10124, American Chemical Society, August 2015 (article)

DOI [BibTex]

Dyna: A Model of Dynamic Human Shape in Motion

Pons-Moll, G., Romero, J., Mahmood, N., Black, M. J.

ACM Transactions on Graphics, (Proc. SIGGRAPH), 34(4):120:1-120:14, ACM, August 2015 (article)

Abstract

To look human, digital full-body avatars need to have soft tissue deformations like those of real people. We learn a model of soft-tissue deformations from examples using a high-resolution 4D capture system and a method that accurately registers a template mesh to sequences of 3D scans. Using over 40,000 scans of ten subjects, we learn how soft tissue motion causes mesh triangles to deform relative to a base 3D body model. Our Dyna model uses a low-dimensional linear subspace to approximate soft-tissue deformation and relates the subspace coefficients to the changing pose of the body. Dyna uses a second-order auto-regressive model that predicts soft-tissue deformations based on previous deformations, the velocity and acceleration of the body, and the angular velocities and accelerations of the limbs. Dyna also models how deformations vary with a person’s body mass index (BMI), producing different deformations for people with different shapes. Dyna realistically represents the dynamics of soft tissue for previously unseen subjects and motions. We provide tools for animators to modify the deformations and apply them to new stylized characters.

pdf preprint video data DOI Project Page Project Page [BibTex]

ps Pons-Moll, G., Romero, J., Mahmood, N., Black, M. J. Dyna: A Model of Dynamic Human Shape in Motion ACM Transactions on Graphics, (Proc. SIGGRAPH), 34(4):120:1-120:14, ACM, August 2015 (article)

pdf preprint video data DOI Project Page Project Page [BibTex]

Micro-fiber arrays with tip coating and transfer method for preparing same

Sitti, M., Washburn, N. R., Glass, P. S., Chung, H.

July 2015, US Patent 9,079,215 (patent)

Abstract

Present invention describes a patterned and coated micro- and nano-scale fibers elastomeric material for enhanced adhesion in wet or dry environments. A multi-step fabrication process including optical lithography, micromolding, polymer synthesis, dipping, stamping, and photopolymerization is described to produce uniform arrays of micron-scale fibers with mushroom-shaped tips coated with a thin layer of an intrinsically adhesive synthetic polymer, such as lightly crosslinked p(DMA-co-MEA).

[BibTex]

pi Sitti, M., Washburn, N. R., Glass, P. S., Chung, H. Micro-fiber arrays with tip coating and transfer method for preparing same July 2015, US Patent 9,079,215 (patent)

[BibTex]

The Informed Sampler: A Discriminative Approach to Bayesian Inference in Generative Computer Vision Models

Jampani, V., Nowozin, S., Loper, M., Gehler, P. V.

In Special Issue on Generative Models in Computer Vision and Medical Imaging, 136, pages: 32-44, Elsevier, July 2015 (inproceedings)

Abstract

Computer vision is hard because of a large variability in lighting, shape, and texture; in addition the image signal is non-additive due to occlusion. Generative models promised to account for this variability by accurately modelling the image formation process as a function of latent variables with prior beliefs. Bayesian posterior inference could then, in principle, explain the observation. While intuitively appealing, generative models for computer vision have largely failed to deliver on that promise due to the difficulty of posterior inference. As a result the community has favored efficient discriminative approaches. We still believe in the usefulness of generative models in computer vision, but argue that we need to leverage existing discriminative or even heuristic computer vision methods. We implement this idea in a principled way in our informed sampler and in careful experiments demonstrate it on challenging models which contain renderer programs as their components. The informed sampler, using simple discriminative proposals based on existing computer vision technology achieves dramatic improvements in inference. Our approach enables a new richness in generative models that was out of reach with existing inference technology.

arXiv-preprint pdf DOI Project Page [BibTex]

ps Jampani, V., Nowozin, S., Loper, M., Gehler, P. V. The Informed Sampler: A Discriminative Approach to Bayesian Inference in Generative Computer Vision Models In Special Issue on Generative Models in Computer Vision and Medical Imaging, 136, pages: 32-44, Elsevier, July 2015 (inproceedings)

arXiv-preprint pdf DOI Project Page [BibTex]

Linking Objects to Actions: Encoding of Target Object and Grasping Strategy in Primate Ventral Premotor Cortex

Vargas-Irwin, C. E., Franquemont, L., Black, M. J., Donoghue, J. P.

Journal of Neuroscience, 35(30):10888-10897, July 2015 (article)

Abstract

Neural activity in ventral premotor cortex (PMv) has been associated with the process of matching perceived objects with the motor commands needed to grasp them. It remains unclear how PMv networks can flexibly link percepts of objects affording multiple grasp options into a final desired hand action. Here, we use a relational encoding approach to track the functional state of PMv neuronal ensembles in macaque monkeys through the process of passive viewing, grip planning, and grasping movement execution. We used objects affording multiple possible grip strategies. The task included separate instructed delay periods for object presentation and grip instruction. This approach allowed us to distinguish responses elicited by the visual presentation of the objects from those associated with selecting a given motor plan for grasping. We show that PMv continuously incorporates information related to object shape and grip strategy as it becomes available, revealing a transition from a set of ensemble states initially most closely related to objects, to a new set of ensemble patterns reflecting unique object-grip combinations. These results suggest that PMv dynamically combines percepts, gradually navigating toward activity patterns associated with specific volitional actions, rather than directly mapping perceptual object properties onto categorical grip representations. Our results support the idea that PMv is part of a network that dynamically computes motor plans from perceptual information. Significance Statement: The present work demonstrates that the activity of groups of neurons in primate ventral premotor cortex reflects information related to visually presented objects, as well as the motor strategy used to grasp them, linking individual objects to multiple possible grips. PMv could provide useful control signals for neuroprosthetic assistive devices designed to interact with objects in a flexible way.

publisher link DOI Project Page [BibTex]

ps Vargas-Irwin, C. E., Franquemont, L., Black, M. J., Donoghue, J. P. Linking Objects to Actions: Encoding of Target Object and Grasping Strategy in Primate Ventral Premotor Cortex Journal of Neuroscience, 35(30):10888-10897, July 2015 (article)

publisher link DOI Project Page [BibTex]

pH-taxis of biohybrid microsystems

Zhuang, J., Carlsen, R. W., Sitti, M.

Scientific reports, 5, Nature Publishing Group, June 2015 (article)

Abstract

The last decade has seen an increasing number of studies developing bacteria and other cell-integrated biohybrid microsystems. However, the highly stochastic motion of these microsystems severely limits their potential use. Here, we present a method that exploits the pH sensing of flagellated bacteria to realize robust drift control of multi-bacteria propelled microrobots. Under three specifically configured pH gradients, we demonstrate that the microrobots exhibit both unidirectional and bidirectional pH-tactic behaviors, which are also observed in free-swimming bacteria. From trajectory analysis, we find that the swimming direction and speed biases are two major factors that contribute to their tactic drift motion. The motion analysis of microrobots also sheds light on the propulsion dynamics of the flagellated bacteria as bioactuators. It is expected that similar driving mechanisms are shared among pH-taxis, chemotaxis, and thermotaxis. By identifying the mechanism that drives the tactic behavior of bacteria-propelled microsystems, this study opens up an avenue towards improving the control of biohybrid microsystems. Furthermore, assuming that it is possible to tune the preferred pH of bioactuators by genetic engineering, these biohybrid microsystems could potentially be applied to sense the pH gradient induced by cancerous cells in stagnant fluids inside human body and realize targeted drug delivery.

DOI Project Page [BibTex]

pi Zhuang, J., Carlsen, R. W., Sitti, M. pH-taxis of biohybrid microsystems Scientific reports, 5, Nature Publishing Group, June 2015 (article)

DOI Project Page [BibTex]

The Stitched Puppet: A Graphical Model of 3D Human Shape and Pose

Zuffi, S., Black, M. J.

In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2015), pages: 3537-3546, June 2015 (inproceedings)

Abstract

We propose a new 3D model of the human body that is both realistic and part-based. The body is represented by a graphical model in which nodes of the graph correspond to body parts that can independently translate and rotate in 3D as well as deform to capture pose-dependent shape variations. Pairwise potentials define a “stitching cost” for pulling the limbs apart, giving rise to the stitched puppet model (SPM). Unlike existing realistic 3D body models, the distributed representation facilitates inference by allowing the model to more effectively explore the space of poses, much like existing 2D pictorial structures models. We infer pose and body shape using a form of particle-based max-product belief propagation. This gives the SPM the realism of recent 3D body models with the computational advantages of part-based models. We apply the SPM to two challenging problems involving estimating human shape and pose from 3D data. The first is the FAUST mesh alignment challenge (http://faust.is.tue.mpg.de/), where ours is the first method to successfully align all 3D meshes. The second involves estimating pose and shape from crude visual hull representations of complex body movements.

pdf Extended Abstract poster code/project video DOI Project Page [BibTex]

ps Zuffi, S., Black, M. J. The Stitched Puppet: A Graphical Model of 3D Human Shape and Pose In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2015), pages: 3537-3546, June 2015 (inproceedings)

pdf Extended Abstract poster code/project video DOI Project Page [BibTex]

Displets: Resolving Stereo Ambiguities using Object Knowledge

Güney, F., Geiger, A.

In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) 2015, pages: 4165-4175, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), June 2015 (inproceedings)

Abstract

Stereo techniques have witnessed tremendous progress over the last decades, yet some aspects of the problem still remain challenging today. Striking examples are reflecting and textureless surfaces which cannot easily be recovered using traditional local regularizers. In this paper, we therefore propose to regularize over larger distances using object-category specific disparity proposals (displets) which we sample using inverse graphics techniques based on a sparse disparity estimate and a semantic segmentation of the image. The proposed displets encode the fact that objects of certain categories are not arbitrarily shaped but typically exhibit regular structures. We integrate them as non-local regularizer for the challenging object class 'car' into a superpixel based CRF framework and demonstrate its benefits on the KITTI stereo evaluation.

avg ps

pdf abstract suppmat [BibTex]

avg ps Güney, F., Geiger, A. Displets: Resolving Stereo Ambiguities using Object Knowledge In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) 2015, pages: 4165-4175, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), June 2015 (inproceedings)

pdf abstract suppmat [BibTex]

Object Scene Flow for Autonomous Vehicles

Menze, M., Geiger, A.

In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) 2015, pages: 3061-3070, IEEE, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), June 2015 (inproceedings)

Abstract

This paper proposes a novel model and dataset for 3D scene flow estimation with an application to autonomous driving. Taking advantage of the fact that outdoor scenes often decompose into a small number of independently moving objects, we represent each element in the scene by its rigid motion parameters and each superpixel by a 3D plane as well as an index to the corresponding object. This minimal representation increases robustness and leads to a discrete-continuous CRF where the data term decomposes into pairwise potentials between superpixels and objects. Moreover, our model intrinsically segments the scene into its constituting dynamic components. We demonstrate the performance of our model on existing benchmarks as well as a novel realistic dataset with scene flow ground truth. We obtain this dataset by annotating 400 dynamic scenes from the KITTI raw data collection using detailed 3D CAD models for all vehicles in motion. Our experiments also reveal novel challenges which can't be handled by existing methods.

avg ps

pdf abstract suppmat DOI [BibTex]

avg ps Menze, M., Geiger, A. Object Scene Flow for Autonomous Vehicles In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) 2015, pages: 3061-3070, IEEE, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), June 2015 (inproceedings)

pdf abstract suppmat DOI [BibTex]

Pose-Conditioned Joint Angle Limits for 3D Human Pose Reconstruction

Akhter, I., Black, M. J.

In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2015), pages: 1446-1455, June 2015 (inproceedings)

Abstract

The estimation of 3D human pose from 2D joint locations is central to many vision problems involving the analysis of people in images and video. To address the fact that the problem is inherently ill posed, many methods impose a prior over human poses. Unfortunately these priors admit invalid poses because they do not model how joint-limits vary with pose. Here we make two key contributions. First, we collected a motion capture dataset that explores a wide range of human poses. From this we learn a pose-dependent model of joint limits that forms our prior. The dataset and the prior will be made publicly available. Second, we define a general parameterization of body pose and a new, multistage, method to estimate 3D pose from 2D joint locations that uses an over-complete dictionary of human poses. Our method shows good generalization while avoiding impossible poses. We quantitatively compare our method with recent work and show state-of-the-art results on 2D to 3D pose estimation using the CMU mocap dataset. We also show superior results on manual annotations on real images and automatic part-based detections on the Leeds sports pose dataset.

pdf Extended Abstract video project/data/code poster DOI Project Page Project Page [BibTex]

ps Akhter, I., Black, M. J. Pose-Conditioned Joint Angle Limits for 3D Human Pose Reconstruction In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2015), pages: 1446-1455, June 2015 (inproceedings)

pdf Extended Abstract video project/data/code poster DOI Project Page Project Page [BibTex]

Efficient Sparse-to-Dense Optical Flow Estimation using a Learned Basis and Layers

Wulff, J., Black, M. J.

In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2015), pages: 120-130, June 2015 (inproceedings)

Abstract

We address the elusive goal of estimating optical flow both accurately and efficiently by adopting a sparse-to-dense approach. Given a set of sparse matches, we regress to dense optical flow using a learned set of full-frame basis flow fields. We learn the principal components of natural flow fields using flow computed from four Hollywood movies. Optical flow fields are then compactly approximated as a weighted sum of the basis flow fields. Our new PCA-Flow algorithm robustly estimates these weights from sparse feature matches. The method runs in under 300ms/frame on the MPI-Sintel dataset using a single CPU and is more accurate and significantly faster than popular methods such as LDOF and Classic+NL. The results, however, are too smooth for some applications. Consequently, we develop a novel sparse layered flow method in which each layer is represented by PCA-flow. Unlike existing layered methods, estimation is fast because it uses only sparse matches. We combine information from different layers into a dense flow field using an image-aware MRF. The resulting PCA-Layers method runs in 3.6s/frame, is significantly more accurate than PCA-flow and achieves state-of-the-art performance in occluded regions on MPI-Sintel.

pdf Extended Abstract Supplemental Material Poster Code Project Page Project Page [BibTex]

ps Wulff, J., Black, M. J. Efficient Sparse-to-Dense Optical Flow Estimation using a Learned Basis and Layers In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2015), pages: 120-130, June 2015 (inproceedings)

pdf Extended Abstract Supplemental Material Poster Code Project Page Project Page [BibTex]

Permutohedral Lattice CNNs

Kiefel, M., Jampani, V., Gehler, P. V.

In ICLR Workshop Track, ICLR, May 2015 (inproceedings)

Abstract

This paper presents a convolutional layer that is able to process sparse input features. As an example, for image recognition problems this allows an efficient filtering of signals that do not lie on a dense grid (like pixel position), but of more general features (such as color values). The presented algorithm makes use of the permutohedral lattice data structure. The permutohedral lattice was introduced to efficiently implement a bilateral filter, a commonly used image processing operation. Its use allows for a generalization of the convolution type found in current (spatial) convolutional network architectures.

ei ps

pdf link (url) [BibTex]

ei ps Kiefel, M., Jampani, V., Gehler, P. V. Permutohedral Lattice CNNs In ICLR Workshop Track, ICLR, May 2015 (inproceedings)

pdf link (url) [BibTex]

Fiberbot: A miniature crawling robot using a directional fibrillar pad

Han, Y., Marvi, H., Sitti, M.

In Robotics and Automation (ICRA), 2015 IEEE International Conference on, pages: 3122-3127, May 2015 (inproceedings)

Abstract

Vibration-driven locomotion has been widely used for crawling robot studies. Such robots usually have a vibration motor as the actuator and a fibrillar structure for providing directional friction on the substrate. However, there has not been any studies about the effect of fiber structure on robot crawling performance. In this paper, we develop Fiberbot, a custom made mini vibration robot, for studying the effect of fiber angle on robot velocity, steering, and climbing performance. It is known that the friction force with and against fibers depends on the fiber angle. Thus, we first present a new fabrication method for making millimeter scale fibers at a wide range of angles. We then show that using 30° angle fibers that have the highest friction anisotropy (ratio of backward to forward friction force) among the other fibers we fabricated in this study, Fiberbot speed on glass increases to 13.8±0.4 cm/s (compared to ν = 0.6±0.1 cm/s using vertical fibers). We also demonstrate that the locomotion direction of Fiberbot depends on the tilting direction of fibers and we can steer the robot by rotating the fiber pad. Fiberbot could also climb on glass at inclinations of up to 10° when equipped with fibers of high friction anisotropy. We show that adding a rigid tail to the robot it can climb on glass at 25° inclines. Moreover, the robot is able to crawl on rough surfaces such as wood (ν = 10.0±0.2 cm/s using 30° fiber pad). Fiberbot, a low-cost vibration robot equipped with a custom-designed fiber pad with steering and climbing capabilities could be used for studies on collective behavior on a wide range of topographies as well as search and exploratory missions.

DOI [BibTex]

pi Han, Y., Marvi, H., Sitti, M. Fiberbot: A miniature crawling robot using a directional fibrillar pad In Robotics and Automation (ICRA), 2015 IEEE International Conference on, pages: 3122-3127, May 2015 (inproceedings)

DOI [BibTex]

Platform design and tethered flight of a motor-driven flapping-wing system

Hines, L., Colmenares, D., Sitti, M.

In Robotics and Automation (ICRA), 2015 IEEE International Conference on, pages: 5838-5845, May 2015 (inproceedings)

Abstract

In this work, we examine two design modifications to a tethered motor-driven flapping-wing system. Previously, we had demonstrated a simple mechanism utilizing a linear transmission for resonant operation and direct drive of the wing flapping angle for control. The initial two-wing system had a weight of 2.7 grams and a maximum lift-to-weight ratio of 1.4. While capable of vertical takeoff, in open-loop flight it demonstrated instability and pitch oscillations at the wing flapping frequency, leading to flight times of only a few wing strokes. Here the effect of vertical wing offset as well as an alternative multi-wing layout is investigated and experimentally tested with newly constructed prototypes. With only a change in vertical wing offset, stable open-loop flight of the two-wing flapping system is shown to be theoretically possible, but difficult to achieve with our current design and operating parameters. Both of the new two and four-wing systems, however, prove capable of flying to the end of the tether, with the four-wing system prototype eliminating disruptive wing beat oscillations.

DOI [BibTex]

pi Hines, L., Colmenares, D., Sitti, M. Platform design and tethered flight of a motor-driven flapping-wing system In Robotics and Automation (ICRA), 2015 IEEE International Conference on, pages: 5838-5845, May 2015 (inproceedings)

DOI [BibTex]

Structural optimization for flexure-based parallel mechanisms–Towards achieving optimal dynamic and stiffness properties

Lum, G. Z., Teo, T. J., Yeo, S. H., Yang, G., Sitti, M.

Precision Engineering, 42, pages: 195-207, Elsevier, May 2015 (article)

Abstract

Flexure-based parallel mechanisms (FPMs) are a type of compliant mechanisms that consist of a rigid end-effector that is articulated by several parallel, flexible limbs (a.k.a. sub-chains). Existing design methods can enhance the FPMs’ dynamic and stiffness properties by conducting a size optimization on their sub-chains. A similar optimization process, however, was not performed for their sub-chains’ topology, and this may severely limit the benefits of a size optimization. Thus, this paper proposes to use a structural optimization approach to synthesize and optimize the topology, shape and size of the FPMs’ sub-chains. The benefits of this approach are demonstrated via the design and development of a planar X − Y − θz FPM. A prototype of this FPM was evaluated experimentally to have a large workspace of 1.2 mm × 1.2 mm × 6°, a fundamental natural frequency of 102 Hz, and stiffness ratios that are greater than 120. The achieved properties show significant improvement over existing 3-degrees-of-freedom compliant mechanisms that can deflect more than 0.5 mm and 0.5°. These compliant mechanisms typically have stiffness ratios that are less than 60 and a fundamental natural frequency that is less than 45 Hz.

DOI [BibTex]

pi Lum, G. Z., Teo, T. J., Yeo, S. H., Yang, G., Sitti, M. Structural optimization for flexure-based parallel mechanisms–Towards achieving optimal dynamic and stiffness properties Precision Engineering, 42, pages: 195-207, Elsevier, May 2015 (article)

DOI [BibTex]

Consensus Message Passing for Layered Graphical Models

Jampani, V., Eslami, S. M. A., Tarlow, D., Kohli, P., Winn, J.

In Eighteenth International Conference on Artificial Intelligence and Statistics (AISTATS), 38, pages: 425-433, JMLR Workshop and Conference Proceedings, Eighteenth International Conference on Artificial Intelligence and Statistics, May 2015 (inproceedings)

Abstract

Generative models provide a powerful framework for probabilistic reasoning. However, in many domains their use has been hampered by the practical difficulties of inference. This is particularly the case in computer vision, where models of the imaging process tend to be large, loopy and layered. For this reason bottom-up conditional models have traditionally dominated in such domains. We find that widely-used, general-purpose message passing inference algorithms such as Expectation Propagation (EP) and Variational Message Passing (VMP) fail on the simplest of vision models. With these models in mind, we introduce a modification to message passing that learns to exploit their layered structure by passing 'consensus' messages that guide inference towards good solutions. Experiments on a variety of problems show that the proposed technique leads to significantly more accurate inference results, not only when compared to standard EP and VMP, but also when compared to competitive bottom-up conditional models.

online pdf supplementary link (url) [BibTex]

ps Jampani, V., Eslami, S. M. A., Tarlow, D., Kohli, P., Winn, J. Consensus Message Passing for Layered Graphical Models In Eighteenth International Conference on Artificial Intelligence and Statistics (AISTATS), 38, pages: 425-433, JMLR Workshop and Conference Proceedings, Eighteenth International Conference on Artificial Intelligence and Statistics, May 2015 (inproceedings)

online pdf supplementary link (url) [BibTex]

Shape Models of the Human Body for Distributed Inference

Zuffi, S.

Brown University, May 2015 (phdthesis)

Abstract

In this thesis we address the problem of building shape models of the human body, in 2D and 3D, which are realistic and efficient to use. We focus our efforts on the human body, which is highly articulated and has interesting shape variations, but the approaches we present here can be applied to generic deformable and articulated objects. To address efficiency, we constrain our models to be part-based and have a tree-structured representation with pairwise relationships between connected parts. This allows the application of methods for distributed inference based on message passing. To address realism, we exploit recent advances in computer graphics that represent the human body with statistical shape models learned from 3D scans. We introduce two articulated body models, a 2D model, named Deformable Structures (DS), which is a contour-based model parameterized for 2D pose and projected shape, and a 3D model, named Stitchable Puppet (SP), which is a mesh-based model parameterized for 3D pose, pose-dependent deformations and intrinsic body shape. We have successfully applied the models to interesting and challenging problems in computer vision and computer graphics, namely pose estimation from static images, pose estimation from video sequences, pose and shape estimation from 3D scan data. This advances the state of the art in human pose and shape estimation and suggests that carefully dened realistic models can be important for computer vision. More work at the intersection of vision and graphics is thus encouraged.

PDF [BibTex]

ps Zuffi, S. Shape Models of the Human Body for Distributed Inference Brown University, May 2015 (phdthesis)

PDF [BibTex]

Controlled surface topography regulates collective 3D migration by epithelial–mesenchymal composite embryonic tissues

Song, J., Shawky, J. H., Kim, Y., Hazar, M., LeDuc, P. R., Sitti, M., Davidson, L. A.

Biomaterials, 58, pages: 1-9, Elsevier, April 2015 (article)

Abstract

Cells in tissues encounter a range of physical cues as they migrate. Probing single cell and collective migratory responses to physically defined three-dimensional (3D) microenvironments and the factors that modulate those responses are critical to understanding how tissue migration is regulated during development, regeneration, and cancer. One key physical factor that regulates cell migration is topography. Most studies on surface topography and cell mechanics have been carried out with single migratory cells, yet little is known about the spreading and motility response of 3D complex multi-cellular tissues to topographical cues. Here, we examine the response to complex topographical cues of microsurgically isolated tissue explants composed of epithelial and mesenchymal cell layers from naturally 3D organized embryos of the aquatic frog Xenopus laevis. We control topography using fabricated micropost arrays (MPAs) and investigate the collective 3D migration of these multi-cellular systems in these MPAs. We find that the topography regulates both collective and individual cell migration and that dense MPAs reduce but do not eliminate tissue spreading. By modulating cell size through the cell cycle inhibitor Mitomycin C or the spacing of the MPAs we uncover how 3D topographical cues disrupt collective cell migration. We find surface topography can direct both single cell motility and tissue spreading, altering tissue-scale processes that enable efficient conversion of single cell motility into collective movement.

DOI [BibTex]

pi Song, J., Shawky, J. H., Kim, Y., Hazar, M., LeDuc, P. R., Sitti, M., Davidson, L. A. Controlled surface topography regulates collective 3D migration by epithelial–mesenchymal composite embryonic tissues Biomaterials, 58, pages: 1-9, Elsevier, April 2015 (article)

DOI [BibTex]

Transfer Printing of Metallic Microstructures on Adhesion-Promoting Hydrogel Substrates

Wu, H., Sariola, V., Zhu, C., Zhao, J., Sitti, M., Bettinger, C. J.

Advanced Materials, 27(22):3398-3404, April 2015 (article)

Abstract

Fabrication schemes that integrate inorganic microstructures with hydrogel substrates are essential for advancing flexible electronics. A transfer printing process that is made possible through the design and synthesis of adhesion-promoting hydrogels as target substrates is reported. This fabrication technique may advance ultracompliant electronics by melding microfabricated structures with swollen hydrogel substrates.

DOI [BibTex]

pi Wu, H., Sariola, V., Zhu, C., Zhao, J., Sitti, M., Bettinger, C. J. Transfer Printing of Metallic Microstructures on Adhesion-Promoting Hydrogel Substrates Advanced Materials, 27(22):3398-3404, April 2015 (article)

DOI [BibTex]

Dry adhesives and methods for making dry adhesives

Sitti, M., Murphy, M., Aksak, B.

March 2015, US Patent App. 14/625,162 (patent)

Abstract

Dry adhesives and methods for forming dry adhesives. A method of forming a dry adhesive structure on a substrate, comprises: forming a template backing layer of energy sensitive material on the substrate; forming a template layer of energy sensitive material on the template backing layer; exposing the template layer to a predetermined pattern of energy; removing a portion of the template layer related to the predetermined pattern of energy, and leaving a template structure formed from energy sensitive material and connected to the substrate via the template backing layer.

[BibTex]

pi Sitti, M., Murphy, M., Aksak, B. Dry adhesives and methods for making dry adhesives March 2015, US Patent App. 14/625,162 (patent)

[BibTex]

Biomedical applications of untethered mobile milli/microrobots

Sitti, M., Ceylan, H., Hu, W., Giltinan, J., Turan, M., Yim, S., Diller, E.

Proceedings of the IEEE, 103(2):205-224, IEEE, March 2015 (article)

Abstract

Untethered robots miniaturized to the length scale of millimeter and below attract growing attention for the prospect of transforming many aspects of health care and bioengineering. As the robot size goes down to the order of a single cell, previously inaccessible body sites would become available for high-resolution in situ and in vivo manipulations. This unprecedented direct access would enable an extensive range of minimally invasive medical operations. Here, we provide a comprehensive review of the current advances in biomedical untethered mobile milli/microrobots. We put a special emphasis on the potential impacts of biomedical microrobots in the near future. Finally, we discuss the existing challenges and emerging concepts associated with designing such a miniaturized robot for operation inside a biological environment for biomedical applications.

DOI [BibTex]

pi Sitti, M., Ceylan, H., Hu, W., Giltinan, J., Turan, M., Yim, S., Diller, E. Biomedical applications of untethered mobile milli/microrobots Proceedings of the IEEE, 103(2):205-224, IEEE, March 2015 (article)

DOI [BibTex]

Multi-view and 3D Deformable Part Models

Pepik, B., Stark, M., Gehler, P., Schiele, B.

Pattern Analysis and Machine Intelligence, 37(11):14, IEEE, March 2015 (article)

Abstract

As objects are inherently 3-dimensional, they have been modeled in 3D in the early days of computer vision. Due to the ambiguities arising from mapping 2D features to 3D models, 3D object representations have been neglected and 2D feature-based models are the predominant paradigm in object detection nowadays. While such models have achieved outstanding bounding box detection performance, they come with limited expressiveness, as they are clearly limited in their capability of reasoning about 3D shape or viewpoints. In this work, we bring the worlds of 3D and 2D object representations closer, by building an object detector which leverages the expressive power of 3D object representations while at the same time can be robustly matched to image evidence. To that end, we gradually extend the successful deformable part model [1] to include viewpoint information and part-level 3D geometry information, resulting in several different models with different level of expressiveness. We end up with a 3D object model, consisting of multiple object parts represented in 3D and a continuous appearance model. We experimentally verify that our models, while providing richer object hypotheses than the 2D object models, provide consistently better joint object localization and viewpoint estimation than the state-of-the-art multi-view and 3D object detectors on various benchmarks (KITTI [2], 3D object classes [3], Pascal3D+ [4], Pascal VOC 2007 [5], EPFL multi-view cars [6]).

DOI Project Page [BibTex]

ps Pepik, B., Stark, M., Gehler, P., Schiele, B. Multi-view and 3D Deformable Part Models Pattern Analysis and Machine Intelligence, 37(11):14, IEEE, March 2015 (article)

DOI Project Page [BibTex]

From Scans to Models: Registration of 3D Human Shapes Exploiting Texture Information

Bogo, F.

University of Padova, March 2015 (phdthesis)

Abstract

New scanning technologies are increasing the importance of 3D mesh data, and of algorithms that can reliably register meshes obtained from multiple scans. Surface registration is important e.g. for building full 3D models from partial scans, identifying and tracking objects in a 3D scene, creating statistical shape models. Human body registration is particularly important for many applications, ranging from biomedicine and robotics to the production of movies and video games; but obtaining accurate and reliable registrations is challenging, given the articulated, non-rigidly deformable structure of the human body. In this thesis, we tackle the problem of 3D human body registration. We start by analyzing the current state of the art, and find that: a) most registration techniques rely only on geometric information, which is ambiguous on flat surface areas; b) there is a lack of adequate datasets and benchmarks in the field. We address both issues. Our contribution is threefold. First, we present a model-based registration technique for human meshes that combines geometry and surface texture information to provide highly accurate mesh-to-mesh correspondences. Our approach estimates scene lighting and surface albedo, and uses the albedo to construct a high-resolution textured 3D body model that is brought into registration with multi-camera image data using a robust matching term. Second, by leveraging our technique, we present FAUST (Fine Alignment Using Scan Texture), a novel dataset collecting 300 high-resolution scans of 10 people in a wide range of poses. FAUST is the first dataset providing both real scans and automatically computed, reliable "ground-truth" correspondences between them. Third, we explore possible uses of our approach in dermatology. By combining our registration technique with a melanocytic lesion segmentation algorithm, we propose a system that automatically detects new or evolving lesions over almost the entire body surface, thus helping dermatologists identify potential melanomas. We conclude this thesis investigating the benefits of using texture information to establish frame-to-frame correspondences in dynamic monocular sequences captured with consumer depth cameras. We outline a novel approach to reconstruct realistic body shape and appearance models from dynamic human performances, and show preliminary results on challenging sequences captured with a Kinect.

[BibTex]

ps Bogo, F. From Scans to Models: Registration of 3D Human Shapes Exploiting Texture Information University of Padova, March 2015 (phdthesis)

[BibTex]

Long Range Motion Estimation and Applications

Sevilla-Lara, L.

Long Range Motion Estimation and Applications, University of Massachusetts Amherst, University of Massachusetts Amherst, Febuary 2015 (phdthesis)

Abstract

Finding correspondences between images underlies many computer vision problems, such as optical flow, tracking, stereovision and alignment. Finding these correspondences involves formulating a matching function and optimizing it. This optimization process is often gradient descent, which avoids exhaustive search, but relies on the assumption of being in the basin of attraction of the right local minimum. This is often the case when the displacement is small, and current methods obtain very accurate results for small motions. However, when the motion is large and the matching function is bumpy this assumption is less likely to be true. One traditional way of avoiding this abruptness is to smooth the matching function spatially by blurring the images. As the displacement becomes larger, the amount of blur required to smooth the matching function becomes also larger. This averaging of pixels leads to a loss of detail in the image. Therefore, there is a trade-off between the size of the objects that can be tracked and the displacement that can be captured. In this thesis we address the basic problem of increasing the size of the basin of attraction in a matching function. We use an image descriptor called distribution fields (DFs). By blurring the images in DF space instead of in pixel space, we in- crease the size of the basin attraction with respect to traditional methods. We show competitive results using DFs both in object tracking and optical flow. Finally we demonstrate an application of capturing large motions for temporal video stitching.

[BibTex]

ps Sevilla-Lara, L. Long Range Motion Estimation and Applications Long Range Motion Estimation and Applications, University of Massachusetts Amherst, University of Massachusetts Amherst, Febuary 2015 (phdthesis)

[BibTex]

Collective 3D Migration of Embryonic Epithelial Mesenchymal Composite Tissues are Regulated by Surface Topology

Song, J., Shawky, J., Kim, Y. T., Hazar, M., Sitti, M., LeDuc, P. R., Davidson, L. A.

Biophysical Journal, 108(2):455a, Elsevier, January 2015 (article)

Abstract

Cells in tissues encounter a range of physical cues as they migrate. Probing single cell and collective migratory responses to physically defined three-dimensional (3D) microenvironments and the factors that modulate those responses are critical to understanding how tissue migration is regulated during development, regeneration, and cancer. One key physical factor that regulates cell migration is topology. Most studies on surface topology and cell mechanics have been carried out with single migratory cells, yet little is known about the spreading and motility response of 3D complex multicellular tissues to topological cues. Here, we examine the behaviors of microsurgically isolated tissue explants composed of epithelial and mesenchymal cell layers from naturally 3D organized embryos of the aquatic frog Xenopus laevis to complex topological cues. We control topology using fabricated micropost arrays (MPAs) with different diameters (e.g., different spacing gaps) and investigate the collective 3D migration of these multicellular systems in these MPAs. Our topographical controlled approach for cellular application enables us to achieve a high degree of control over micropost positioning and geometry via simple, accurate, and repeatable microfabrication processes. We find that the topology regulates both collective and individual cell migration and that dense MPAs reduce but do not eliminate tissue spreading. By modulating cell size through the cell cycle inhibitor Mitomycin C or the spacing within MPAs we discover a role for topology in disrupting collective enhancement of cell migration. We find 3D topological cues can direct both single cell motility and tissue spreading, altering tissue-scale processes that enable efficient conversion of single cell motility into collective movement.

DOI [BibTex]

pi Song, J., Shawky, J., Kim, Y. T., Hazar, M., Sitti, M., LeDuc, P. R., Davidson, L. A. Collective 3D Migration of Embryonic Epithelial Mesenchymal Composite Tissues are Regulated by Surface Topology Biophysical Journal, 108(2):455a, Elsevier, January 2015 (article)

DOI [BibTex]

Three-dimensional heterogeneous assembly of coded microgels using an untethered mobile microgripper

Chung, S. E., Dong, X., Sitti, M.

Lab on a Chip, 15(7):1667-1676, Royal Society of Chemistry, January 2015 (article)

Abstract

Three-dimensional (3D) heterogeneous assembly of coded microgels in enclosed aquatic environments is demonstrated using a remotely actuated and controlled magnetic microgripper by a customized electromagnetic coil system. The microgripper uses different ‘stick–slip’ and ‘rolling’ locomotion in 2D and also levitation in 3D by magnetic gradient-based pulling force. This enables the microrobot to precisely manipulate each microgel by controlling its position and orientation in all x–y–z directions. Our microrobotic assembly method broke the barrier of limitation on the number of assembled microgel layers, because it enabled precise 3D levitation of the microgripper. We used the gripper to assemble microgels that had been coded with different colours and shapes onto prefabricated polymeric microposts. This eliminates the need for extra secondary cross-linking to fix the final construct. We demonstrated assembly of microgels on a single micropost up to ten layers. By increasing the number and changing the distribution of the posts, complex heterogeneous microsystems were possible to construct in 3D.

DOI Project Page [BibTex]

pi Chung, S. E., Dong, X., Sitti, M. Three-dimensional heterogeneous assembly of coded microgels using an untethered mobile microgripper Lab on a Chip, 15(7):1667-1676, Royal Society of Chemistry, January 2015 (article)

DOI Project Page [BibTex]

Integrating mechanism synthesis and topological optimization technique for stiffness-oriented design of a three degrees-of-freedom flexure-based parallel mechanism

Lum, G. Z., Teo, T. J., Yang, G., Yeo, S. H., Sitti, M.

Precision Engineering, 39, pages: 125-133, Elsevier, January 2015 (article)

Abstract

This paper introduces a new design approach to synthesize multiple degrees-of-freedom (DOF) flexure-based parallel mechanism (FPM). Termed as an integrated design approach, it is a systematic design methodology, which integrates both classical mechanism synthesis and modern topology optimization technique, to deliver an optimized multi-DOF FPM. This design approach is separated into two levels. At sub-chain level, a novel topology optimization technique, which uses the classical linkage mechanisms as DNA seeds, is used to synthesize the compliant joints or limbs. At configuration level, the optimal compliant joints are used to form the parallel limbs of the multi-DOF FPM and another stage of optimization was conducted to determine the optimal space distribution between these compliant joints so as to generate a multi-DOF FPM with optimized stiffness characteristic. In this paper, the design of a 3-DOF planar motion FPM was used to demonstrate the effectiveness and accuracy of this proposed design approach.

DOI [BibTex]

pi Lum, G. Z., Teo, T. J., Yang, G., Yeo, S. H., Sitti, M. Integrating mechanism synthesis and topological optimization technique for stiffness-oriented design of a three degrees-of-freedom flexure-based parallel mechanism Precision Engineering, 39, pages: 125-133, Elsevier, January 2015 (article)

DOI [BibTex]

Actively controlled fibrillar friction surfaces

Marvi, H, Han, Y, Sitti, M

Applied Physics Letters, 106(5):051602, AIP Publishing, January 2015 (article)

Abstract

In this letter, we propose a technique by which we can actively adjust frictional properties of elastic fibrillar structures in different directions. Using a mesh attached to a two degree-of-freedom linear stage, we controlled the active length and the tilt angle of fibers, independently. Thus, we were able to achieve desired levels of friction forces in different directions and significantly improve passive friction anisotropies observed in the same fiber arrays. The proposed technique would allow us to readily control the friction anisotropy and the friction magnitude of fibrillar structures in any planar direction.

DOI [BibTex]

pi Marvi, H, Han, Y, Sitti, M Actively controlled fibrillar friction surfaces Applied Physics Letters, 106(5):051602, AIP Publishing, January 2015 (article)

DOI [BibTex]

Spike train SIMilarity Space (SSIMS): A framework for single neuron and ensemble data analysis

Vargas-Irwin, C. E., Brandman, D. M., Zimmermann, J. B., Donoghue, J. P., Black, M. J.

Neural Computation, 27(1):1-31, MIT Press, January 2015 (article)

Abstract

We present a method to evaluate the relative similarity of neural spiking patterns by combining spike train distance metrics with dimensionality reduction. Spike train distance metrics provide an estimate of similarity between activity patterns at multiple temporal resolutions. Vectors of pair-wise distances are used to represent the intrinsic relationships between multiple activity patterns at the level of single units or neuronal ensembles. Dimensionality reduction is then used to project the data into concise representations suitable for clustering analysis as well as exploratory visualization. Algorithm performance and robustness are evaluated using multielectrode ensemble activity data recorded in behaving primates. We demonstrate how Spike train SIMilarity Space (SSIMS) analysis captures the relationship between goal directions for an 8-directional reaching task and successfully segregates grasp types in a 3D grasping task in the absence of kinematic information. The algorithm enables exploration of virtually any type of neural spiking (time series) data, providing similarity-based clustering of neural activity states with minimal assumptions about potential information encoding models.

pdf: publisher site pdf: author's proof DOI Project Page [BibTex]

ps Vargas-Irwin, C. E., Brandman, D. M., Zimmermann, J. B., Donoghue, J. P., Black, M. J. Spike train SIMilarity Space (SSIMS): A framework for single neuron and ensemble data analysis Neural Computation, 27(1):1-31, MIT Press, January 2015 (article)

pdf: publisher site pdf: author's proof DOI Project Page [BibTex]

Efficient Facade Segmentation using Auto-Context

Jampani, V., Gadde, R., Gehler, P. V.

In Applications of Computer Vision (WACV), 2015 IEEE Winter Conference on, pages: 1038-1045, IEEE, WACV,, January 2015 (inproceedings)

Abstract

In this paper we propose a system for the problem of facade segmentation. Building facades are highly structured images and consequently most methods that have been proposed for this problem, aim to make use of this strong prior information. We are describing a system that is almost domain independent and consists of standard segmentation methods. A sequence of boosted decision trees is stacked using auto-context features and learned using the stacked generalization technique. We find that this, albeit standard, technique performs better, or equals, all previous published empirical results on all available facade benchmark datasets. The proposed method is simple to implement, easy to extend, and very efficient at test time inference.

website pdf supplementary IEEE page link (url) DOI Project Page [BibTex]

ps Jampani, V., Gadde, R., Gehler, P. V. Efficient Facade Segmentation using Auto-Context In Applications of Computer Vision (WACV), 2015 IEEE Winter Conference on, pages: 1038-1045, IEEE, WACV,, January 2015 (inproceedings)

website pdf supplementary IEEE page link (url) DOI Project Page [BibTex]

FlowCap: 2D Human Pose from Optical Flow

Romero, J., Loper, M., Black, M. J.

In Pattern Recognition, Proc. 37th German Conference on Pattern Recognition (GCPR), LNCS 9358, pages: 412-423, Springer, GCPR, 2015 (inproceedings)

Abstract

We estimate 2D human pose from video using only optical flow. The key insight is that dense optical flow can provide information about 2D body pose. Like range data, flow is largely invariant to appearance but unlike depth it can be directly computed from monocular video. We demonstrate that body parts can be detected from dense flow using the same random forest approach used by the Microsoft Kinect. Unlike range data, however, when people stop moving, there is no optical flow and they effectively disappear. To address this, our FlowCap method uses a Kalman filter to propagate body part positions and ve- locities over time and a regression method to predict 2D body pose from part centers. No range sensor is required and FlowCap estimates 2D human pose from monocular video sources containing human motion. Such sources include hand-held phone cameras and archival television video. We demonstrate 2D body pose estimation in a range of scenarios and show that the method works with real-time optical flow. The results suggest that optical flow shares invariances with range data that, when complemented with tracking, make it valuable for pose estimation.

video pdf preprint Project Page Project Page [BibTex]

ps Romero, J., Loper, M., Black, M. J. FlowCap: 2D Human Pose from Optical Flow In Pattern Recognition, Proc. 37th German Conference on Pattern Recognition (GCPR), LNCS 9358, pages: 412-423, Springer, GCPR, 2015 (inproceedings)

video pdf preprint Project Page Project Page [BibTex]

Metric Regression Forests for Correspondence Estimation

Pons-Moll, G., Taylor, J., Shotton, J., Hertzmann, A., Fitzgibbon, A.

International Journal of Computer Vision, pages: 1-13, 2015 (article)

springer PDF Project Page [BibTex]

ps Pons-Moll, G., Taylor, J., Shotton, J., Hertzmann, A., Fitzgibbon, A. Metric Regression Forests for Correspondence Estimation International Journal of Computer Vision, pages: 1-13, 2015 (article)

springer PDF Project Page [BibTex]

Joint 3D Object and Layout Inference from a single RGB-D Image

(Best Paper Award)

Geiger, A., Wang, C.

In German Conference on Pattern Recognition (GCPR), 9358, pages: 183-195, Lecture Notes in Computer Science, Springer International Publishing, 2015 (inproceedings)

Abstract

Inferring 3D objects and the layout of indoor scenes from a single RGB-D image captured with a Kinect camera is a challenging task. Towards this goal, we propose a high-order graphical model and jointly reason about the layout, objects and superpixels in the image. In contrast to existing holistic approaches, our model leverages detailed 3D geometry using inverse graphics and explicitly enforces occlusion and visibility constraints for respecting scene properties and projective geometry. We cast the task as MAP inference in a factor graph and solve it efficiently using message passing. We evaluate our method with respect to several baselines on the challenging NYUv2 indoor dataset using 21 object categories. Our experiments demonstrate that the proposed method is able to infer scenes with a large degree of clutter and occlusions.

avg ps

pdf suppmat video project DOI [BibTex]

avg ps Geiger, A., Wang, C. Joint 3D Object and Layout Inference from a single RGB-D Image In German Conference on Pattern Recognition (GCPR), 9358, pages: 183-195, Lecture Notes in Computer Science, Springer International Publishing, 2015 (inproceedings)

pdf suppmat video project DOI [BibTex]

3D Object Class Detection in the Wild

Pepik, B., Stark, M., Gehler, P., Ritschel, T., Schiele, B.

In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2015 (inproceedings)

Project Page [BibTex]

ps Pepik, B., Stark, M., Gehler, P., Ritschel, T., Schiele, B. 3D Object Class Detection in the Wild In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2015 (inproceedings)

Project Page [BibTex]

Discrete Optimization for Optical Flow

Menze, M., Heipke, C., Geiger, A.

In German Conference on Pattern Recognition (GCPR), 9358, pages: 16-28, Springer International Publishing, 2015 (inproceedings)

Abstract

We propose to look at large-displacement optical flow from a discrete point of view. Motivated by the observation that sub-pixel accuracy is easily obtained given pixel-accurate optical flow, we conjecture that computing the integral part is the hardest piece of the problem. Consequently, we formulate optical flow estimation as a discrete inference problem in a conditional random field, followed by sub-pixel refinement. Naive discretization of the 2D flow space, however, is intractable due to the resulting size of the label set. In this paper, we therefore investigate three different strategies, each able to reduce computation and memory demands by several orders of magnitude. Their combination allows us to estimate large-displacement optical flow both accurately and efficiently and demonstrates the potential of discrete optimization for optical flow. We obtain state-of-the-art performance on MPI Sintel and KITTI.

avg ps

pdf suppmat project DOI [BibTex]

avg ps Menze, M., Heipke, C., Geiger, A. Discrete Optimization for Optical Flow In German Conference on Pattern Recognition (GCPR), 9358, pages: 16-28, Springer International Publishing, 2015 (inproceedings)

pdf suppmat project DOI [BibTex]

Joint 3D Estimation of Vehicles and Scene Flow

Menze, M., Heipke, C., Geiger, A.

In Proc. of the ISPRS Workshop on Image Sequence Analysis (ISA), 2015 (inproceedings)

Abstract

Three-dimensional reconstruction of dynamic scenes is an important prerequisite for applications like mobile robotics or autonomous driving. While much progress has been made in recent years, imaging conditions in natural outdoor environments are still very challenging for current reconstruction and recognition methods. In this paper, we propose a novel unified approach which reasons jointly about 3D scene flow as well as the pose, shape and motion of vehicles in the scene. Towards this goal, we incorporate a deformable CAD model into a slanted-plane conditional random field for scene flow estimation and enforce shape consistency between the rendered 3D models and the parameters of all superpixels in the image. The association of superpixels to objects is established by an index variable which implicitly enables model selection. We evaluate our approach on the challenging KITTI scene flow dataset in terms of object and scene flow estimation. Our results provide a prove of concept and demonstrate the usefulness of our method.

avg ps

PDF [BibTex]

avg ps Menze, M., Heipke, C., Geiger, A. Joint 3D Estimation of Vehicles and Scene Flow In Proc. of the ISPRS Workshop on Image Sequence Analysis (ISA), 2015 (inproceedings)

PDF [BibTex]

MPI Papers

Departments

Research Groups

Publication Type

Year

2015

2015

Latest News

Links

Contact Us