Header logo is


2018


Thumb xl screenshot 2018 5 9 swimming back and forth using planar flagellar propulsion at low reynolds numbers   khalil   2018   adv ...
Swimming Back and Forth Using Planar Flagellar Propulsion at Low Reynolds Numbers

Khalil, I. S. M., Tabak, A. F., Hamed, Y., Mitwally, M. E., Tawakol, M., Klingner, A., Sitti, M.

Advanced Science, 5(2):1700461, 2018 (article)

Abstract
Abstract Peritrichously flagellated Escherichia coli swim back and forth by wrapping their flagella together in a helical bundle. However, other monotrichous bacteria cannot swim back and forth with a single flagellum and planar wave propagation. Quantifying this observation, a magnetically driven soft two‐tailed microrobot capable of reversing its swimming direction without making a U‐turn trajectory or actively modifying the direction of wave propagation is designed and developed. The microrobot contains magnetic microparticles within the polymer matrix of its head and consists of two collinear, unequal, and opposite ultrathin tails. It is driven and steered using a uniform magnetic field along the direction of motion with a sinusoidally varying orthogonal component. Distinct reversal frequencies that enable selective and independent excitation of the first or the second tail of the microrobot based on their tail length ratio are found. While the first tail provides a propulsive force below one of the reversal frequencies, the second is almost passive, and the net propulsive force achieves flagellated motion along one direction. On the other hand, the second tail achieves flagellated propulsion along the opposite direction above the reversal frequency.

pi

link (url) DOI [BibTex]

2018


link (url) DOI [BibTex]


Thumb xl universal custom complex magnetic spring design methodology
Universal Custom Complex Magnetic Spring Design Methodology

Woodward, M. A., Sitti, M.

IEEE Transactions on Magnetics, 54(1):1-13, October 2018 (article)

Abstract
A design methodology is presented for creating custom complex magnetic springs through the design of force-displacement curves. This methodology results in a magnet configuration, which will produce a desired force-displacement relationship. Initially, the problem is formulated and solved as a system of linear equations. Then, given the limited likelihood of a single solution being feasibly manufactured, key parameters of the solution are extracted and varied to create a family of solutions. Finally, these solutions are refined using numerical optimization. Given the properties of magnets, this methodology can create any well-defined function of force versus displacement and is model-independent. To demonstrate this flexibility, a number of example magnetic springs are designed; one of which, designed for use in a jumping-gliding robot's shape memory alloy actuated clutch, is manufactured and experimentally characterized. Due to the scaling of magnetic forces, the displacement region which these magnetic springs are most applicable is that of millimeters and below. However, this region is well situated for miniature robots and smart material actuators, where a tailored magnetic spring, designed to compliment a component, can enhance its performance while adding new functionality. The methodology is also expendable to variable interactions and multi-dimensional magnetic field design.

pi

DOI [BibTex]

DOI [BibTex]


Thumb xl sevillagcpr
On the Integration of Optical Flow and Action Recognition

Sevilla-Lara, L., Liao, Y., Güney, F., Jampani, V., Geiger, A., Black, M. J.

In German Conference on Pattern Recognition (GCPR), LNCS 11269, pages: 281-297, Springer, Cham, October 2018 (inproceedings)

Abstract
Most of the top performing action recognition methods use optical flow as a "black box" input. Here we take a deeper look at the combination of flow and action recognition, and investigate why optical flow is helpful, what makes a flow method good for action recognition, and how we can make it better. In particular, we investigate the impact of different flow algorithms and input transformations to better understand how these affect a state-of-the-art action recognition method. Furthermore, we fine tune two neural-network flow methods end-to-end on the most widely used action recognition dataset (UCF101). Based on these experiments, we make the following five observations: 1) optical flow is useful for action recognition because it is invariant to appearance, 2) optical flow methods are optimized to minimize end-point-error (EPE), but the EPE of current methods is not well correlated with action recognition performance, 3) for the flow methods tested, accuracy at boundaries and at small displacements is most correlated with action recognition performance, 4) training optical flow to minimize classification error instead of minimizing EPE improves recognition performance, and 5) optical flow learned for the task of action recognition differs from traditional optical flow especially inside the human body and at the boundary of the body. These observations may encourage optical flow researchers to look beyond EPE as a goal and guide action recognition researchers to seek better motion cues, leading to a tighter integration of the optical flow and action recognition communities.

avg ps

arXiv DOI [BibTex]

arXiv DOI [BibTex]


Thumb xl iros18
Towards Robust Visual Odometry with a Multi-Camera System

Liu, P., Geppert, M., Heng, L., Sattler, T., Geiger, A., Pollefeys, M.

In International Conference on Intelligent Robots and Systems (IROS) 2018, International Conference on Intelligent Robots and Systems, October 2018 (inproceedings)

Abstract
We present a visual odometry (VO) algorithm for a multi-camera system and robust operation in challenging environments. Our algorithm consists of a pose tracker and a local mapper. The tracker estimates the current pose by minimizing photometric errors between the most recent keyframe and the current frame. The mapper initializes the depths of all sampled feature points using plane-sweeping stereo. To reduce pose drift, a sliding window optimizer is used to refine poses and structure jointly. Our formulation is flexible enough to support an arbitrary number of stereo cameras. We evaluate our algorithm thoroughly on five datasets. The datasets were captured in different conditions: daytime, night-time with near-infrared (NIR) illumination and night-time without NIR illumination. Experimental results show that a multi-camera setup makes the VO more robust to challenging environments, especially night-time conditions, in which a single stereo configuration fails easily due to the lack of features.

avg

pdf Project Page [BibTex]

pdf Project Page [BibTex]


Thumb xl ianeccv18
Learning Priors for Semantic 3D Reconstruction

Cherabier, I., Schönberger, J., Oswald, M., Pollefeys, M., Geiger, A.

In Computer Vision – ECCV 2018, Springer International Publishing, Cham, September 2018 (inproceedings)

Abstract
We present a novel semantic 3D reconstruction framework which embeds variational regularization into a neural network. Our network performs a fixed number of unrolled multi-scale optimization iterations with shared interaction weights. In contrast to existing variational methods for semantic 3D reconstruction, our model is end-to-end trainable and captures more complex dependencies between the semantic labels and the 3D geometry. Compared to previous learning-based approaches to 3D reconstruction, we integrate powerful long-range dependencies using variational coarse-to-fine optimization. As a result, our network architecture requires only a moderate number of parameters while keeping a high level of expressiveness which enables learning from very little data. Experiments on real and synthetic datasets demonstrate that our network achieves higher accuracy compared to a purely variational approach while at the same time requiring two orders of magnitude less iterations to converge. Moreover, our approach handles ten times more semantic class labels using the same computational resources.

avg

pdf suppmat Project Page Video DOI Project Page [BibTex]

pdf suppmat Project Page Video DOI Project Page [BibTex]


no image
Discovering and Teaching Optimal Planning Strategies

Lieder, F., Callaway, F., Krueger, P. M., Das, P., Griffiths, T. L., Gul, S.

In The 14th biannual conference of the German Society for Cognitive Science, GK, September 2018 (inproceedings)

re

Project Page [BibTex]

Project Page [BibTex]


Thumb xl joeleccv18
Unsupervised Learning of Multi-Frame Optical Flow with Occlusions

Janai, J., Güney, F., Ranjan, A., Black, M. J., Geiger, A.

In European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, vol 11220, pages: 713-731, Springer, Cham, September 2018 (inproceedings)

avg ps

pdf suppmat Video Project Page DOI Project Page [BibTex]

pdf suppmat Video Project Page DOI Project Page [BibTex]


no image
Discovering Rational Heuristics for Risky Choice

Gul, S., Krueger, P. M., Callaway, F., Griffiths, T. L., Lieder, F.

The 14th biannual conference of the German Society for Cognitive Science, GK, The 14th biannual conference of the German Society for Cognitive Science, GK, September 2018 (conference)

re

Project Page [BibTex]

Project Page [BibTex]


Thumb xl beneccv18
SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images

Coors, B., Condurache, A. P., Geiger, A.

European Conference on Computer Vision (ECCV), September 2018 (conference)

Abstract
Omnidirectional cameras offer great benefits over classical cameras wherever a wide field of view is essential, such as in virtual reality applications or in autonomous robots. Unfortunately, standard convolutional neural networks are not well suited for this scenario as the natural projection surface is a sphere which cannot be unwrapped to a plane without introducing significant distortions, particularly in the polar regions. In this work, we present SphereNet, a novel deep learning framework which encodes invariance against such distortions explicitly into convolutional neural networks. Towards this goal, SphereNet adapts the sampling locations of the convolutional filters, effectively reversing distortions, and wraps the filters around the sphere. By building on regular convolutions, SphereNet enables the transfer of existing perspective convolutional neural network models to the omnidirectional case. We demonstrate the effectiveness of our method on the tasks of image classification and object detection, exploiting two newly created semi-synthetic and real-world omnidirectional datasets.

avg

pdf suppmat Project Page [BibTex]


Thumb xl teaser image
Programmable collective behavior in dynamically self-assembled mobile microrobotic swarms

B Yigit, , Y Alapan, , Sitti, M.

Advanced Science, July 2018 (article)

Abstract
Collective control of mobile microrobotic swarms is indispensable for their potential high-impact applications in targeted drug delivery, medical diagnostics, parallel micromanipulation, and environmental sensing and remediation. Lack of on-board computational and sensing capabilities in current microrobotic systems necessitates use of physical interactions among individual microrobots for local physical communication and cooperation. Here, we show that mobile microrobotic swarms with well-defined collective behavior can be designed by engineering magnetic interactions among individual units. Microrobots, consisting of a linear chain of self-assembled magnetic microparticles, locomote on surfaces in response to a precessing magnetic field. Control over the direction of precessing magnetic field allows engineering attractive and repulsive interactions among microrobots and, thus, collective order with well-defined spatial organization and parallel operation over macroscale distances (~ 1 cm). These microrobotic swarms can be guided through confined spaces, while preserving microrobot morphology and function. These swarms can further achieve directional transport of large cargoes on surfaces and small cargoes in bulk fluids. Described design approach, exploiting physical interactions among individual robots, enables facile and rapid formation of self-organized and reconfigurable microrobotic swarms with programmable collective order.

pi

link (url) [BibTex]


Thumb xl picture1
3D-Printed Biodegradable Microswimmer for Drug Delivery and Targeted Cell Labeling

Hakan Ceylan, , I. Ceren Yasa, , Oncay Yasa, , Ahmet Fatih Tabak, , Joshua Giltinan, , Sitti, M.

bioRxiv, pages: 379024, July 2018 (article)

Abstract
Miniaturization of interventional medical devices can leverage minimally invasive technologies by enabling operational resolution at cellular length scales with high precision and repeatability. Untethered micron-scale mobile robots can realize this by navigating and performing in hard-to-reach, confined and delicate inner body sites. However, such a complex task requires an integrated design and engineering strategy, where powering, control, environmental sensing, medical functionality and biodegradability need to be considered altogether. The present study reports a hydrogel-based, biodegradable microrobotic swimmer, which is responsive to the changes in its microenvironment for theranostic cargo delivery and release tasks. We design a double-helical magnetic microswimmer of 20 micrometers length, which is 3D-printed with complex geometrical and compositional features. At normal physiological concentrations, matrix metalloproteinase-2 (MMP-2) enzyme can entirely degrade the microswimmer body in 118 h to solubilized non-toxic products. The microswimmer can respond to the pathological concentrations of MMP-2 by swelling and thereby accelerating the release kinetics of the drug payload. Anti-ErbB 2 antibody-tagged magnetic nanoparticles released from the degraded microswimmers serve for targeted labeling of SKBR3 breast cancer cells to realize the potential of medical imaging of local tissue sites following the therapeutic intervention. These results represent a leap forward toward clinical medical microrobots that are capable of sensing, responding to the local pathological information, and performing specific therapeutic and diagnostic tasks as orderly executed operations using their smart composite material architectures.

pi

DOI Project Page [BibTex]


Thumb xl screen shot 2018 06 29 at 4.24.39 pm
Innate turning preference of leaf-cutting ants in the absence of external orientation cues

Endlein, T., Sitti, M.

Journal of Experimental Biology, The Company of Biologists Ltd, June 2018 (article)

Abstract
Many ants use a combination of cues for orientation but how do ants find their way when all external cues are suppressed? Do they walk in a random way or are their movements spatially oriented? Here we show for the first time that leaf-cutting ants (Acromyrmex lundii) have an innate preference of turning counter-clockwise (left) when external cues are precluded. We demonstrated this by allowing individual ants to run freely on the water surface of a newly-developed treadmill. The surface tension supported medium-sized workers but effectively prevented ants from reaching the wall of the vessel, important to avoid wall-following behaviour (thigmotaxis). Most ants ran for minutes on the spot but also slowly turned counter-clockwise in the absence of visual cues. Reconstructing the effectively walked path revealed a looping pattern which could be interpreted as a search strategy. A similar turning bias was shown for groups of ants in a symmetrical Y-maze where twice as many ants chose the left branch in the absence of optical cues. Wall-following behaviour was tested by inserting a coiled tube before the Y-fork. When ants traversed a left-coiled tube, more ants chose the left box and vice versa. Adding visual cues in form of vertical black strips either outside the treadmill or on one branch of the Y-maze led to oriented walks towards the strips. It is suggested that both, the turning bias and the wall-following are employed as search strategies for an unknown environment which can be overridden by visual cues.

pi

link (url) DOI [BibTex]

link (url) DOI [BibTex]


Thumb xl 1
Motility and chemotaxis of bacteria-driven microswimmers fabricated using antigen 43-mediated biotin display

Schauer, O., Mostaghaci, B., Colin, R., Hürtgen, D., Kraus, D., Sitti, M., Sourjik, V.

Scientific Reports, 8(1):9801, Nature Publishing Group, June 2018 (article)

Abstract
Bacteria-driven biohybrid microswimmers (bacteriabots) combine synthetic cargo with motile living bacteria that enable propulsion and steering. Although fabrication and potential use of such bacteriabots have attracted much attention, existing methods of fabrication require an extensive sample preparation that can drastically decrease the viability and motility of bacteria. Moreover, chemotactic behavior of bacteriabots in a liquid medium with chemical gradients has remained largely unclear. To overcome these shortcomings, we designed Escherichia coli to autonomously display biotin on its cell surface via the engineered autotransporter antigen 43 and thus to bind streptavidin-coated cargo. We show that the cargo attachment to these bacteria is greatly enhanced by motility and occurs predominantly at the cell poles, which is greatly beneficial for the fabrication of motile bacteriabots. We further performed a systemic study to understand and optimize the ability of these bacteriabots to follow chemical gradients. We demonstrate that the chemotaxis of bacteriabots is primarily limited by the cargo-dependent reduction of swimming speed and show that the fabrication of bacteriabots using elongated E. coli cells can be used to overcome this limitation.

pi

link (url) DOI [BibTex]

link (url) DOI [BibTex]


Thumb xl 41586 2018 250 fig1 html
Multifunctional ferrofluid-infused surfaces with reconfigurable multiscale topography

Wang, W., Timonen, J. V. I., Carlson, A., Drotlef, D., Zhang, C. T., Kolle, S., Grinthal, A., Wong, T., Hatton, B., Kang, S. H., Kennedy, S., Chi, J., Blough, R. T., Sitti, M., Mahadevan, L., Aizenberg, J.

Nature, June 2018 (article)

Abstract
Developing adaptive materials with geometries that change in response to external stimuli provides fundamental insights into the links between the physical forces involved and the resultant morphologies and creates a foundation for technologically relevant dynamic systems1,2. In particular, reconfigurable surface topography as a means to control interfacial properties 3 has recently been explored using responsive gels 4 , shape-memory polymers 5 , liquid crystals6-8 and hybrid composites9-14, including magnetically active slippery surfaces12-14. However, these designs exhibit a limited range of topographical changes and thus a restricted scope of function. Here we introduce a hierarchical magneto-responsive composite surface, made by infiltrating a ferrofluid into a microstructured matrix (termed ferrofluid-containing liquid-infused porous surfaces, or FLIPS). We demonstrate various topographical reconfigurations at multiple length scales and a broad range of associated emergent behaviours. An applied magnetic-field gradient induces the movement of magnetic nanoparticles suspended in the ferrofluid, which leads to microscale flow of the ferrofluid first above and then within the microstructured surface. This redistribution changes the initially smooth surface of the ferrofluid (which is immobilized by the porous matrix through capillary forces) into various multiscale hierarchical topographies shaped by the size, arrangement and orientation of the confining microstructures in the magnetic field. We analyse the spatial and temporal dynamics of these reconfigurations theoretically and experimentally as a function of the balance between capillary and magnetic pressures15-19 and of the geometric anisotropy of the FLIPS system. Several interesting functions at three different length scales are demonstrated: self-assembly of colloidal particles at the micrometre scale; regulated flow of liquid droplets at the millimetre scale; and switchable adhesion and friction, liquid pumping and removal of biofilms at the centimetre scale. We envision that FLIPS could be used as part of integrated control systems for the manipulation and transport of matter, thermal management, microfluidics and fouling-release materials.

pi

link (url) DOI [BibTex]

link (url) DOI [BibTex]


Thumb xl selfsensing
Self-Sensing Paper Actuators Based on Graphite–Carbon Nanotube Hybrid Films

Amjadi, M., Sitti, M.

Advanced Science, pages: 1800239, May 2018 (article)

Abstract
Abstract Soft actuators have demonstrated potential in a range of applications, including soft robotics, artificial muscles, and biomimetic devices. However, the majority of current soft actuators suffer from the lack of real-time sensory feedback, prohibiting their effective sensing and multitask function. Here, a promising strategy is reported to design bilayer electrothermal actuators capable of simultaneous actuation and sensation (i.e., self-sensing actuators), merely through two input electric terminals. Decoupled electrothermal stimulation and strain sensation is achieved by the optimal combination of graphite microparticles and carbon nanotubes (CNTs) in the form of hybrid films. By finely tuning the charge transport properties of hybrid films, the signal-to-noise ratio (SNR) of self-sensing actuators is remarkably enhanced to over 66. As a result, self-sensing actuators can actively track their displacement and distinguish the touch of soft and hard objects.

pi

link (url) DOI Project Page [BibTex]


Thumb xl andrease teaser 2
Robust Dense Mapping for Large-Scale Dynamic Environments

Barsan, I. A., Liu, P., Pollefeys, M., Geiger, A.

In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) 2018, IEEE, International Conference on Robotics and Automation, May 2018 (inproceedings)

Abstract
We present a stereo-based dense mapping algorithm for large-scale dynamic urban environments. In contrast to other existing methods, we simultaneously reconstruct the static background, the moving objects, and the potentially moving but currently stationary objects separately, which is desirable for high-level mobile robotic tasks such as path planning in crowded environments. We use both instance-aware semantic segmentation and sparse scene flow to classify objects as either background, moving, or potentially moving, thereby ensuring that the system is able to model objects with the potential to transition from static to dynamic, such as parked cars. Given camera poses estimated from visual odometry, both the background and the (potentially) moving objects are reconstructed separately by fusing the depth maps computed from the stereo input. In addition to visual odometry, sparse scene flow is also used to estimate the 3D motions of the detected moving objects, in order to reconstruct them accurately. A map pruning technique is further developed to improve reconstruction accuracy and reduce memory consumption, leading to increased scalability. We evaluate our system thoroughly on the well-known KITTI dataset. Our system is capable of running on a PC at approximately 2.5Hz, with the primary bottleneck being the instance-aware semantic segmentation, which is a limitation we hope to address in future work.

avg

pdf Video Project Page Project Page [BibTex]

pdf Video Project Page Project Page [BibTex]


Thumb xl screenshot 2018 05 18 16 38 40
Learning 3D Shape Completion under Weak Supervision

Stutz, D., Geiger, A.

Arxiv, May 2018 (article)

Abstract
We address the problem of 3D shape completion from sparse and noisy point clouds, a fundamental problem in computer vision and robotics. Recent approaches are either data-driven or learning-based: Data-driven approaches rely on a shape model whose parameters are optimized to fit the observations; Learning-based approaches, in contrast, avoid the expensive optimization step by learning to directly predict complete shapes from incomplete observations in a fully-supervised setting. However, full supervision is often not available in practice. In this work, we propose a weakly-supervised learning-based approach to 3D shape completion which neither requires slow optimization nor direct supervision. While we also learn a shape prior on synthetic data, we amortize, i.e., learn, maximum likelihood fitting using deep neural networks resulting in efficient shape completion without sacrificing accuracy. On synthetic benchmarks based on ShapeNet and ModelNet as well as on real robotics data from KITTI and Kinect, we demonstrate that the proposed amortized maximum likelihood approach is able to compete with fully supervised baselines and outperforms data-driven approaches, while requiring less supervision and being significantly faster.

avg

PDF Project Page Project Page [BibTex]


Thumb xl f1.large
Soft erythrocyte-based bacterial microswimmers for cargo delivery

Alapan, Y., Yasa, O., Schauer, O., Giltinan, J., Tabak, A. F., Sourjik, V., Sitti, M.

Science Robotics, 3(17):eaar4423, Science Robotics, April 2018 (article)

Abstract
Bacteria-propelled biohybrid microswimmers have recently shown to be able to actively transport and deliver cargos encapsulated into their synthetic constructs to specific regions locally. However, usage of synthetic materials as cargo carriers can result in inferior performance in load-carrying efficiency, biocompatibility, and biodegradability, impeding clinical translation of biohybrid microswimmers. Here, we report construction and external guidance of bacteria-driven microswimmers using red blood cells (RBCs; erythrocytes) as autologous cargo carriers for active and guided drug delivery. Multifunctional biohybrid microswimmers were fabricated by attachment of RBCs [loaded with anticancer doxorubicin drug molecules and superparamagnetic iron oxide nanoparticles (SPIONs)] to bioengineered motile bacteria, Escherichia coli MG1655, via biotin-avidin-biotin binding complex. Autonomous and on-board propulsion of biohybrid microswimmers was provided by bacteria, and their external magnetic guidance was enabled by SPIONs loaded into the RBCs. Furthermore, bacteria-driven RBC microswimmers displayed preserved deformability and attachment stability even after squeezing in microchannels smaller than their sizes, as in the case of bare RBCs. In addition, an on-demand light-activated hyperthermia termination switch was engineered for RBC microswimmers to control bacteria population after operations. RBCs, as biological and autologous cargo carriers in the biohybrid microswimmers, offer notable advantages in stability, deformability, biocompatibility, and biodegradability over synthetic cargo-carrier materials. The biohybrid microswimmer design presented here transforms RBCs from passive cargo carriers into active and guidable cargo carriers toward targeted drug and other cargo delivery applications in medicine.

pi

link (url) DOI Project Page Project Page [BibTex]

link (url) DOI Project Page Project Page [BibTex]


no image
Miniature soft robots – road to the clinic

Sitti, M.

Nature Reviews Materials, April 2018 (article)

Abstract
Soft small robots offer the opportunity to non-invasively access human tissue to perform medical operations and deliver drugs; however, challenges in materials design, biocompatibility and function control remain to be overcome for soft robots to reach the clinic.

pi

link (url) DOI [BibTex]

link (url) DOI [BibTex]


Thumb xl nl 2018 001642 0005
Wrinkling Instability and Adhesion of a Highly Bendable Gallium Oxide Nanofilm Encapsulating a Liquid-Gallium Droplet

Yunusa, M., Amador, G. J., Drotlef, D., Sitti, M.

Nano Letters, 18(4):2498-2504, March 2018 (article)

Abstract
The wrinkling and interfacial adhesion mechanics of a gallium-oxide nanofilm encapsulating a liquid-gallium droplet are presented. The native oxide nanofilm provides mechanical stability by preventing the flow of the liquid metal. We show how a crumpled oxide skin a few nanometers thick behaves akin to a highly bendable elastic nanofilm under ambient conditions. Upon compression, a wrinkling instability emerges at the contact interface to relieve the applied stress. As the load is further increased, radial wrinkles evolve, and, eventually, the oxide nanofilm ruptures. The observed wrinkling closely resembles the instability experienced by nanofilms under axisymmetric loading, thus providing further insights into the behaviors of elastic nanofilms. Moreover, the mechanical attributes of the oxide skin enable high surface conformation by exhibiting liquid-like behavior. We measured an adhesion energy of 0.238 ± 0.008 J m–2 between a liquid-gallium droplet and smooth flat glass, which is close to the measurements of thin-sheet nanomaterials such as graphene on silicon dioxide.

pi

link (url) DOI [BibTex]


Thumb xl screenshot 2018 5 9 1803 01048 pdf
Magnetic-Visual Sensor Fusion-based Dense 3D Reconstruction and Localization for Endoscopic Capsule Robots

Turan, M., Almalioglu, Y., Ornek, E. P., Araujo, H., Yanik, M. F., Sitti, M.

ArXiv e-prints, March 2018 (article)

Abstract
Reliable and real-time 3D reconstruction and localization functionality is a crucial prerequisite for the navigation of actively controlled capsule endoscopic robots as an emerging, minimally invasive diagnostic and therapeutic technology for use in the gastrointestinal (GI) tract. In this study, we propose a fully dense, non-rigidly deformable, strictly real-time, intraoperative map fusion approach for actively controlled endoscopic capsule robot applications which combines magnetic and vision-based localization, with non-rigid deformations based frame-to-model map fusion. The performance of the proposed method is demonstrated using four different ex-vivo porcine stomach models. Across different trajectories of varying speed and complexity, and four different endoscopic cameras, the root mean square surface reconstruction errors 1.58 to 2.17 cm.

pi

link (url) [BibTex]

link (url) [BibTex]


Thumb xl screenshot 2018 5 9 1803 01047 pdf
Unsupervised Odometry and Depth Learning for Endoscopic Capsule Robots

Turan, M., Ornek, E. P., Ibrahimli, N., Giracoglu, C., Almalioglu, Y., Yanik, M. F., Sitti, M.

ArXiv e-prints, March 2018 (article)

Abstract
In the last decade, many medical companies and research groups have tried to convert passive capsule endoscopes as an emerging and minimally invasive diagnostic technology into actively steerable endoscopic capsule robots which will provide more intuitive disease detection, targeted drug delivery and biopsy-like operations in the gastrointestinal(GI) tract. In this study, we introduce a fully unsupervised, real-time odometry and depth learner for monocular endoscopic capsule robots. We establish the supervision by warping view sequences and assigning the re-projection minimization to the loss function, which we adopt in multi-view pose estimation and single-view depth estimation network. Detailed quantitative and qualitative analyses of the proposed framework performed on non-rigidly deformable ex-vivo porcine stomach datasets proves the effectiveness of the method in terms of motion estimation and depth recovery.

pi

link (url) [BibTex]

link (url) [BibTex]


Thumb xl mabi201700377 fig 0001 m
Self‐Folded Hydrogel Tubes for Implantable Muscular Tissue Scaffolds

Vannozzi, L., Yasa, I. C., Ceylan, H., Menciassi, A., Ricotti, L., Sitti, M.

Macromolecular Bioscience, (0), March 2018 (article)

Abstract
Abstract Programming materials with tunable physical and chemical interactions among its components pave the way of generating 3D functional active microsystems with various potential applications in tissue engineering, drug delivery, and soft robotics. Here, the development of a recapitulated fascicle‐like implantable muscle construct by programmed self‐folding of poly(ethylene glycol) diacrylate hydrogels is reported. The system comprises two stacked layers, each with differential swelling degrees, stiffnesses, and thicknesses in 2D, which folds into a 3D tube together. Inside the tubes, muscle cell adhesion and their spatial alignment are controlled. Both skeletal and cardiac muscle cells also exhibit high viability, and cardiac myocytes preserve their contractile function over the course of 7 d. Integration of biological cells with smart, shape‐changing materials could give rise to the development of new cellular constructs for hierarchical tissue assembly, drug testing platforms, and biohybrid actuators that can perform sophisticated tasks.

pi

link (url) DOI [BibTex]

link (url) DOI [BibTex]


Thumb xl 41419 2018 379 fig1 html
Redox metals homeostasis in multiple sclerosis and amyotrophic lateral sclerosis: a review

Sheykhansari, S., Kozielski, K., Bill, J., Sitti, M., Gemmati, D., Zamboni, P., Singh, A. V.

Cell Death \& Disease, 9(3):348, March 2018 (article)

Abstract
The effect of redox metals such as iron and copper on multiple sclerosis and amyotrophic lateral sclerosis has been intensively studied. However, the origin of these disorders remains uncertain. This review article critically describes the physiology of redox metals that produce oxidative stress, which in turn leads to cascades of immunomodulatory alteration of neurons in multiple sclerosis and amyotrophic lateral sclerosis. Iron and copper overload has been well established in motor neurons of these diseases' lesions. On the other hand, the role of other metals like cadmium participating indirectly in the redox cascade of neurobiological mechanism is less studied. In the second part of this review, we focus on this less conspicuous correlation between cadmium as an inactive-redox metal and multiple sclerosis and amyotrophic lateral sclerosis, providing novel treatment modalities and approaches as future prospects.

pi

link (url) DOI [BibTex]

link (url) DOI [BibTex]


Thumb xl cancer cells
Cancer cells biomineralize ionic gold into nanoparticles-microplates via secreting defense proteins with specific gold-binding peptides

Singh, A. V., Jahnke, T., Kishore, V., Park, B., Batuwangala, M., Bill, J., Sitti, M.

Acta Biomaterialia, March 2018 (article)

Abstract
Cancer cells have the capacity to synthesize nanoparticles (NPs). The detailed mechanism of this process is not very well documented. We report the mechanism of biomineralization of aqueous gold chloride into NPs and microplates in the breast-cancer cell line MCF7. Spherical gold NPs are synthesized in these cells in the presence of serum in the culture media by the reduction of HAuCl4. In the absence of serum, the cells exhibit gold microplate formation through seed-mediate growth albeit slower reduction. The structural characteristics of the two types of NPs under different media conditions were confirmed using scanning electron microscopy (SEM); crystallinity and metallic properties were assessed with transmission electron microscopy (TEM) and x-ray photoelectron spectroscopy (XPS). Gold-reducing proteins, related to cell stress initiate the biomineralization of HAuCl4 in cells (under serum free conditions) as confirmed by infrared (IR) spectroscopy. MCF7 cells undergo irreversible replicative senescence when exposed to a high concentration of ionic gold and conversely remain in a dormant reversible quiescent state when exposed to a low gold concentration. The latter cellular state was achievable in the presence of the rho/ROCK inhibitor Y-27632. Proteomic analysis revealed consistent expression of specific proteins under serum and serum-free conditions. A high-throughput proteomic approach to screen gold-reducing proteins and peptide sequences was utilized and validated by quartz crystal microbalance with dissipation (QCM-D). Statement of significance Cancer cells are known to synthesize gold nanoparticles and microstructures, which are promising for bioimaging and other therapeutic applications. However, the detailed mechanism of such biomineralization process is not well understood yet. Herein, we demonstrate that cancer cells exposed to gold ions (grown in serum/serum-free conditions) secrete shock and stress-related proteins with specific gold-binding/reducing polypeptides. Cells undergo reversible senescence and can recover normal physiology when treated with the senescence inhibitor depending on culture condition. The use of mammalian cells as microincubators for synthesis of such particles could have potential influence on their uptake and biocompatibility. This study has important implications for in-situ reduction of ionic gold to anisotropic micro-nanostructures that could be used in-vivo clinical applications and tumor photothermal therapy.

pi

link (url) DOI [BibTex]


Thumb xl screenshot 2018 5 9 1802 00475 pdf
Thermocapillary-driven fluid flow within microchannels

Amador, G. J., Tabak, A. F., Ren, Z., Alapan, Y., Yasa, O., Sitti, M.

ArXiv e-prints, Febuary 2018 (article)

Abstract
Surface tension gradients induce Marangoni flow, which may be exploited for fluid transport. At the micrometer scale, these surface-driven flows can be more significant than those driven by pressure. By introducing fluid-fluid interfaces on the walls of microfluidic channels, we use surface tension gradients to drive bulk fluid flows. The gradients are specifically induced through thermal energy, exploiting the temperature dependence of a fluid-fluid interface to generate thermocapillary flow. In this report, we provide the design concept for a biocompatible, thermocapillary microchannel capable of being powered by solar irradiation. Using temperature gradients on the order of degrees Celsius per centimeter, we achieve fluid velocities on the order of millimeters per second. Following experimental observations, fluid dynamic models, and numerical simulation, we find that the fluid velocity is linearly proportional to the provided temperature gradient, enabling full control of the fluid flow within the microchannels.

pi

link (url) Project Page [BibTex]


Thumb xl 138 2017 905 fig1 html
Sparse-then-dense alignment-based 3D map reconstruction method for endoscopic capsule robots

Turan, M., Pilavci, Y. Y., Ganiyusufoglu, I., Araujo, H., Konukoglu, E., Sitti, M.

Machine Vision and Applications, 29(2):345-359, Febuary 2018 (article)

Abstract
Despite significant progress achieved in the last decade to convert passive capsule endoscopes to actively controllable robots, robotic capsule endoscopy still has some challenges. In particular, a fully dense three-dimensional (3D) map reconstruction of the explored organ remains an unsolved problem. Such a dense map would help doctors detect the locations and sizes of the diseased areas more reliably, resulting in more accurate diagnoses. In this study, we propose a comprehensive medical 3D reconstruction method for endoscopic capsule robots, which is built in a modular fashion including preprocessing, keyframe selection, sparse-then-dense alignment-based pose estimation, bundle fusion, and shading-based 3D reconstruction. A detailed quantitative analysis is performed using a non-rigid esophagus gastroduodenoscopy simulator, four different endoscopic cameras, a magnetically activated soft capsule robot, a sub-millimeter precise optical motion tracker, and a fine-scale 3D optical scanner, whereas qualitative ex-vivo experiments are performed on a porcine pig stomach. To the best of our knowledge, this study is the first complete endoscopic 3D map reconstruction approach containing all of the necessary functionalities for a therapeutically relevant 3D map reconstruction.

pi

link (url) DOI [BibTex]

link (url) DOI [BibTex]


Thumb xl khali1 2801793 large
Independent Actuation of Two-Tailed Microrobots

Khalil, I. S. M., Tabak, A. F., Hamed, Y., Tawakol, M., Klingner, A., Gohary, N. E., Mizaikoff, B., Sitti, M.

IEEE Robotics and Automation Letters, 3(3):1703-1710, Febuary 2018 (article)

Abstract
A soft two-tailed microrobot in low Reynolds number fluids does not achieve forward locomotion by identical tails regardless to its wiggling frequency. If the tails are nonidentical, zero forward locomotion is also observed at specific oscillation frequencies (which we refer to as the reversal frequencies), as the propulsive forces imparted to the fluid by each tail are almost equal in magnitude and opposite in direction. We find distinct reversal frequencies for the two-tailed microrobots based on their tail length ratio. At these frequencies, the microrobot achieves negligible net displacement under the influence of a periodic magnetic field. This observation allows us to fabricate groups of microrobots with tail length ratio of 1.24 ± 0.11, 1.48 ± 0.08, and 1.71 ± 0.09. We demonstrate selective actuation of microrobots based on prior characterization of their reversal frequencies. We also implement simultaneous flagellar propulsion of two microrobots and show that they can be controlled to swim along the same direction and opposite to each other using common periodic magnetic fields. In addition, independent motion control of two microrobots is achieved toward two different reference positions with average steady-state error of 110.1 ± 91.8 μm and 146.9 ± 105.9 μm.

pi

link (url) DOI [BibTex]

link (url) DOI [BibTex]


Thumb xl adma201704530 fig 0002 m
Recent Advances in Wearable Transdermal Delivery Systems

Amjadi, M., Sheykhansari, S., Nelson, B. J., Sitti, M.

Advanced Materials, 30(7):1704530, January 2018 (article)

Abstract
Abstract Wearable transdermal delivery systems have recently received tremendous attention due to their noninvasive, convenient, and prolonged administration of pharmacological agents. Here, the material prospects, fabrication processes, and drug‐release mechanisms of these types of therapeutic delivery systems are critically reviewed. The latest progress in the development of multifunctional wearable devices capable of closed‐loop sensation and drug delivery is also discussed. This survey reveals that wearable transdermal delivery has already made an impact in diverse healthcare applications, while several grand challenges remain.

pi

link (url) DOI [BibTex]

link (url) DOI [BibTex]


Thumb xl 1 s2.0 s092523121731665x gr2 lrg
Deep EndoVO: A recurrent convolutional neural network (RCNN) based visual odometry approach for endoscopic capsule robots

Turan, M., Almalioglu, Y., Araujo, H., Konukoglu, E., Sitti, M.

Neurocomputing, 275, pages: 1861 - 1870, January 2018 (article)

Abstract
Ingestible wireless capsule endoscopy is an emerging minimally invasive diagnostic technology for inspection of the GI tract and diagnosis of a wide range of diseases and pathologies. Medical device companies and many research groups have recently made substantial progresses in converting passive capsule endoscopes to active capsule robots, enabling more accurate, precise, and intuitive detection of the location and size of the diseased areas. Since a reliable real time pose estimation functionality is crucial for actively controlled endoscopic capsule robots, in this study, we propose a monocular visual odometry (VO) method for endoscopic capsule robot operations. Our method lies on the application of the deep recurrent convolutional neural networks (RCNNs) for the visual odometry task, where convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are used for the feature extraction and inference of dynamics across the frames, respectively. Detailed analyses and evaluations made on a real pig stomach dataset proves that our system achieves high translational and rotational accuracies for different types of endoscopic capsule robot trajectories.

pi

link (url) DOI [BibTex]

link (url) DOI [BibTex]


Thumb xl nature25443 f3
Small-scale soft-bodied robot with multimodal locomotion

Hu, W., Lum, G. Z., Mastrangeli, M., Sitti, M.

Nature, 554, pages: 81-85, Nature, January 2018 (article)

Abstract
Untethered small-scale (from several millimetres down to a few micrometres in all dimensions) robots that can non-invasively access confined, enclosed spaces may enable applications in microfactories such as the construction of tissue scaffolds by robotic assembly1, in bioengineering such as single-cell manipulation and biosensing2, and in healthcare3,4,5,6 such as targeted drug delivery4 and minimally invasive surgery3,5. Existing small-scale robots, however, have very limited mobility because they are unable to negotiate obstacles and changes in texture or material in unstructured environments7,8,9,10,11,12,13. Of these small-scale robots, soft robots have greater potential to realize high mobility via multimodal locomotion, because such machines have higher degrees of freedom than their rigid counterparts14,15,16. Here we demonstrate magneto-elastic soft millimetre-scale robots that can swim inside and on the surface of liquids, climb liquid menisci, roll and walk on solid surfaces, jump over obstacles, and crawl within narrow tunnels. These robots can transit reversibly between different liquid and solid terrains, as well as switch between locomotive modes. They can additionally execute pick-and-place and cargo-release tasks. We also present theoretical models to explain how the robots move. Like the large-scale robots that can be used to study locomotion17, these soft small-scale robots could be used to study soft-bodied locomotion produced by small organisms.

pi

link (url) DOI Project Page [BibTex]


Thumb xl adfm201704902 fig 0002 m
Light‐Driven Janus Hollow Mesoporous TiO2–Au Microswimmers

Sridhar, V., Park, B., Sitti, M.

Advanced Functional Materials, 28(5):1704902, January 2018 (article)

Abstract
Abstract Light‐driven microswimmers have garnered attention for their potential use in various applications, such as environmental remediation, hydrogen evolution, and targeted drug delivery. Janus hollow mesoporous TiO2/Au (JHP–TiO2–Au) microswimmers with enhanced swimming speeds under low‐intensity ultraviolet (UV) light are presented. The swimmers show enhanced swimming speeds both in presence and absence of H2O2. The microswimmers move due to self‐electrophoresis when UV light is incident on them. There is a threefold increase in speed of JHP–TiO2–Au microswimmers in comparison with Janus solid TiO2/Au (JS–TiO2–Au) microswimmers. This increase in their speed is due to the increase in surface area of the porous swimmers and their hollow structure. These microswimmers are also made steerable by using a thin Co magnetic layer. They can be used in potential environmental applications for active photocatalytic degradation of methylene blue and targeted active drug delivery of an anticancer drug (doxurobicin) in vitro in H2O2 solution. Their increased speed from the presence of a hollow mesoporous structure is beneficial for future potential applications, such as hydrogen evolution, selective heterogeneous photocatalysis, and targeted cargo delivery.

pi

link (url) DOI Project Page [BibTex]


Thumb xl khali1 2792156 hires
Mechanical Rubbing of Blood Clots Using Helical Robots Under Ultrasound Guidance

Khalil, I. S. M., Mahdy, D., Sharkawy, A. E., Moustafa, R. R., Tabak, A. F., Mitwally, M. E., Hesham, S., Hamdi, N., Klingner, A., Mohamed, A., Sitti, M.

IEEE Robotics and Automation Letters, 3(2):1112-1119, January 2018 (article)

Abstract
A simple way to mitigate the potential negative sideeffects associated with chemical lysis of a blood clot is to tear its fibrin network via mechanical rubbing using a helical robot. Here, we achieve mechanical rubbing of blood clots under ultrasound guidance and using external magnetic actuation. Position of the helical robot is determined using ultrasound feedback and used to control its motion toward the clot, whereas the volume of the clots is estimated simultaneously using visual feedback. We characterize the shear modulus and ultimate shear strength of the blood clots to predict their removal rate during rubbing. Our in vitro experiments show the ability to move the helical robot controllably toward clots using ultrasound feedback with average and maximum errors of 0.84 ± 0.41 and 2.15 mm, respectively, and achieve removal rate of -0.614 ± 0.303 mm3/min at room temperature (25 °C) and -0.482 ± 0.23 mm3/min at body temperature (37 °C), under the influence of two rotating dipole fields at frequency of 35 Hz. We also validate the effectiveness of mechanical rubbing by measuring the number of red blood cells and platelets past the clot. Our measurements show that rubbing achieves cell count of (46 ± 10.9) × 104 cell/ml, whereas the count in the absence of rubbing is (2 ± 1.41) × 104 cell/ml, after 40 min.

pi

link (url) DOI [BibTex]

link (url) DOI [BibTex]


Thumb xl us20180021892a1 20180125 d00000
Method and device for reversibly attaching a phase changing metal to an object

Zhou Ye, G. Z. L. M. S.

US Patent Application US 2018/0021892 A1, January 2018 (patent)

Abstract
A method for reversibly attaching a phase changing metal to an object, the method comprising the steps of: providing a substrate having at least one surface at which the phase changing metal is attached, heating the phase changing metal above a phase changing temperature at which the phase changing metal changes its phase from solid to liquid, bringing the phase changing metal, when the phase changing metal is in the liquid phase or before the phase changing metal is brought into the liquid phase, into contact with the object, permitting the phase changing metal to cool below the phase changing temperature, whereby the phase changing metal becomes solid and the object and the phase changing metal become attached to each other, reheating the phase changing metal above the phase changing temperature to liquefy the phase changing metal, and removing the substrate from the object, with the phase changing metal separating from the object and remaining with the substrate.

pi

US Patent Application Database US Patent Application (PDF) [BibTex]


Thumb xl us20180012693a1 20180111 d00000
Method of fabricating a shape-changeable magentic member, method of producing a shape changeable magnetic member and shape changeable magnetic member

Guo Zhan Lum, Z. Y. M. S.

US Patent Application US 2018/0012693 A1, January 2018 (patent)

Abstract
The present invention relates to a method of fabricating a shape-changeable magnetic member comprising a plurality of segments with each segment being able to be magnetized with a desired magnitude and orientation of magnetization, to a method of producing a shape changeable magnetic member composed of a plurality of segments and to a shape changeable magnetic member.

pi

US Patent Application Database US Patent Application (PDF) [BibTex]


Thumb xl despoina paper teaser
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials

Paschalidou, D., Ulusoy, A. O., Schmitt, C., Gool, L., Geiger, A.

In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2018, 2018 (inproceedings)

Abstract
In this paper, we consider the problem of reconstructing a dense 3D model using images captured from different views. Recent methods based on convolutional neural networks (CNN) allow learning the entire task from data. However, they do not incorporate the physics of image formation such as perspective geometry and occlusion. Instead, classical approaches based on Markov Random Fields (MRF) with ray-potentials explicitly model these physical processes, but they cannot cope with large surface appearance variations across different viewpoints. In this paper, we propose RayNet, which combines the strengths of both frameworks. RayNet integrates a CNN that learns view-invariant feature representations with an MRF that explicitly encodes the physics of perspective projection and occlusion. We train RayNet end-to-end using empirical risk minimization. We thoroughly evaluate our approach on challenging real-world datasets and demonstrate its benefits over a piece-wise trained baseline, hand-crafted models as well as other learning-based approaches.

avg

pdf suppmat Video Project Page code Poster Project Page [BibTex]

pdf suppmat Video Project Page code Poster Project Page [BibTex]


no image
Learning to select computations

Callaway, F., Gul, S., Krueger, P., Griffiths, T. L., Lieder, F.

In Uncertainty in Artificial Intelligence: Proceedings of the Thirty-Fourth Conference, 2018 (inproceedings)

re

Project Page [BibTex]

Project Page [BibTex]


no image
Enhanced Non-Steady Gliding Performance of the MultiMo-Bat through Optimal Airfoil Configuration and Control Strategy

Kim, H., Woodward, M. A., Sitti, M.

In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages: 1382-1388, 2018 (inproceedings)

pi

[BibTex]

[BibTex]


no image
Analysis of Magnetic Interaction in Remotely Controlled Magnetic Devices and Its Application to a Capsule Robot for Drug Delivery

Munoz, F., Alici, G., Zhou, H., Li, W., M. Sitti,

IEEE Transactions on Magnetics, 23(1):298-310, 2018 (article)

pi

[BibTex]

[BibTex]


Thumb xl yiyi paper teaser
Deep Marching Cubes: Learning Explicit Surface Representations

Liao, Y., Donne, S., Geiger, A.

In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2018, 2018 (inproceedings)

Abstract
Existing learning based solutions to 3D surface prediction cannot be trained end-to-end as they operate on intermediate representations (eg, TSDF) from which 3D surface meshes must be extracted in a post-processing step (eg, via the marching cubes algorithm). In this paper, we investigate the problem of end-to-end 3D surface prediction. We first demonstrate that the marching cubes algorithm is not differentiable and propose an alternative differentiable formulation which we insert as a final layer into a 3D convolutional neural network. We further propose a set of loss functions which allow for training our model with sparse point supervision. Our experiments demonstrate that the model allows for predicting sub-voxel accurate 3D shapes of arbitrary topology. Additionally, it learns to complete shapes and to separate an object's inside from its outside even in the presence of sparse and incomplete ground truth. We investigate the benefits of our approach on the task of inferring shapes from 3D point clouds. Our model is flexible and can be combined with a variety of shape encoder and shape inference techniques.

avg

pdf suppmat Video Project Page Poster Project Page [BibTex]

pdf suppmat Video Project Page Poster Project Page [BibTex]


Thumb xl teaser andreas
Semantic Visual Localization

Schönberger, J., Pollefeys, M., Geiger, A., Sattler, T.

In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2018, 2018 (inproceedings)

Abstract
Robust visual localization under a wide range of viewing conditions is a fundamental problem in computer vision. Handling the difficult cases of this problem is not only very challenging but also of high practical relevance, eg, in the context of life-long localization for augmented reality or autonomous robots. In this paper, we propose a novel approach based on a joint 3D geometric and semantic understanding of the world, enabling it to succeed under conditions where previous approaches failed. Our method leverages a novel generative model for descriptor learning, trained on semantic scene completion as an auxiliary task. The resulting 3D descriptors are robust to missing observations by encoding high-level 3D geometric and semantic information. Experiments on several challenging large-scale localization datasets demonstrate reliable localization under extreme viewpoint, illumination, and geometry changes.

avg

pdf suppmat Poster Project Page [BibTex]

pdf suppmat Poster Project Page [BibTex]


Thumb xl hassan teaser paper
Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes

Alhaija, H., Mustikovela, S., Mescheder, L., Geiger, A., Rother, C.

International Journal of Computer Vision (IJCV), 2018, 2018 (article)

Abstract
The success of deep learning in computer vision is based on the availability of large annotated datasets. To lower the need for hand labeled images, virtually rendered 3D worlds have recently gained popularity. Unfortunately, creating realistic 3D content is challenging on its own and requires significant human effort. In this work, we propose an alternative paradigm which combines real and synthetic data for learning semantic instance segmentation and object detection models. Exploiting the fact that not all aspects of the scene are equally important for this task, we propose to augment real-world imagery with virtual objects of the target category. Capturing real-world images at large scale is easy and cheap, and directly provides real background appearances without the need for creating complex 3D models of the environment. We present an efficient procedure to augment these images with virtual objects. In contrast to modeling complete 3D environments, our data augmentation approach requires only a few user interactions in combination with 3D models of the target object category. Leveraging our approach, we introduce a novel dataset of augmented urban driving scenes with 360 degree images that are used as environment maps to create realistic lighting and reflections on rendered objects. We analyze the significance of realistic object placement by comparing manual placement by humans to automatic methods based on semantic scene analysis. This allows us to create composite images which exhibit both realistic background appearance as well as a large number of complex object arrangements. Through an extensive set of experiments, we conclude the right set of parameters to produce augmented data which can maximally enhance the performance of instance segmentation models. Further, we demonstrate the utility of the proposed approach on training standard deep models for semantic instance segmentation and object detection of cars in outdoor driving scenarios. We test the models trained on our augmented data on the KITTI 2015 dataset, which we have annotated with pixel-accurate ground truth, and on the Cityscapes dataset. Our experiments demonstrate that the models trained on augmented imagery generalize better than those trained on fully synthetic data or models trained on limited amounts of annotated real data.

avg

pdf Project Page [BibTex]

pdf Project Page [BibTex]


Thumb xl eigval gradpen
Which Training Methods for GANs do actually Converge?

Mescheder, L., Geiger, A., Nowozin, S.

International Conference on Machine learning (ICML), 2018 (conference)

Abstract
Recent work has shown local convergence of GAN training for absolutely continuous data and generator distributions. In this paper, we show that the requirement of absolute continuity is necessary: we describe a simple yet prototypical counterexample showing that in the more realistic case of distributions that are not absolutely continuous, unregularized GAN training is not always convergent. Furthermore, we discuss regularization strategies that were recently proposed to stabilize GAN training. Our analysis shows that GAN training with instance noise or zero-centered gradient penalties converges. On the other hand, we show that Wasserstein-GANs and WGAN-GP with a finite number of discriminator updates per generator update do not always converge to the equilibrium point. We discuss these results, leading us to a new explanation for the stability problems of GAN training. Based on our analysis, we extend our convergence results to more general GANs and prove local convergence for simplified gradient penalties even if the generator and data distributions lie on lower dimensional manifolds. We find these penalties to work well in practice and use them to learn high-resolution generative image models for a variety of datasets with little hyperparameter tuning.

avg

code video paper supplement slides poster Project Page [BibTex]


no image
Anisotropic Gold Nanostructures: Optimization via in Silico Modeling for Hyperthermia

Singh, A., Jahnke, T., Wang, S., Xiao, Y., Alapan, Y., Kharratian, S., Onbasli, M. C., Kozielski, K., David, H., Richter, G., Bill, J., Laux, P., Luch, A., Sitti, M.

ACS Applied Nano Materials, 1(11):6205-6216, 2018 (article)

pi

[BibTex]

[BibTex]


no image
Collectives of Spinning Mobile Microrobots for Navigation and Object Manipulation at the Air-Water Interface

Wang, W., Kishore, V., Koens, L., Lauga, E., Sitti, M.

In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages: 1-9, 2018 (inproceedings)

pi

[BibTex]

[BibTex]


Thumb xl david paper teaser
Learning 3D Shape Completion from Laser Scan Data with Weak Supervision

Stutz, D., Geiger, A.

In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2018, 2018 (inproceedings)

Abstract
3D shape completion from partial point clouds is a fundamental problem in computer vision and computer graphics. Recent approaches can be characterized as either data-driven or learning-based. Data-driven approaches rely on a shape model whose parameters are optimized to fit the observations. Learning-based approaches, in contrast, avoid the expensive optimization step and instead directly predict the complete shape from the incomplete observations using deep neural networks. However, full supervision is required which is often not available in practice. In this work, we propose a weakly-supervised learning-based approach to 3D shape completion which neither requires slow optimization nor direct supervision. While we also learn a shape prior on synthetic data, we amortize, ie, learn, maximum likelihood fitting using deep neural networks resulting in efficient shape completion without sacrificing accuracy. Tackling 3D shape completion of cars on ShapeNet and KITTI, we demonstrate that the proposed amortized maximum likelihood approach is able to compete with a fully supervised baseline and a state-of-the-art data-driven approach while being significantly faster. On ModelNet, we additionally show that the approach is able to generalize to other object categories as well.

avg

pdf suppmat Project Page Poster Project Page [BibTex]

pdf suppmat Project Page Poster Project Page [BibTex]