The post is a open... defense of open sourcing. Examples ranging from Wikipedia, to Linux, to the open source machine learning software resulting from the Netflix competition are clear examples of the power of the wisdom of the "crowds" (well, the technically wise one, to be exact). Without sacrificing economic return, as shown by Linux's Red Hat.

The article has a nice quote by Chris Anderson, founder of DIY Drones and author of the upcoming Makers: The New Industrial Revolution:

In general, the Open Source Hardware innovation model of the DIY Drones dev teams is designed to beat proprietary innovation models in the speed and cost dimensions, but not necessarily in features or performance. We've developed several autopilot systems with an average dev time of one year and a cost of $0 (all volunteer labor). Comparable commercial systems can take 3-5 years and up to several million dollars. Our systems are designed to approach theirs in performance at 1/10th to 1/100th the cost, which is only possible through an open innovation model.

Indeed, I think in some newer domains, such as multicopters, this may go even further. We're already seeing the open source projects starting to pass the proprietary ones in features, if not performance. And as commodity sensor and processor technology gets cheaper and more capable, thanks to smartphones, the amateur designs could pull ahead of the pro ones in performance, too, simply because they can use new technology sooner. This is why DARPA used our model to create UAV Forge, which is designed to let community-based UAV technology compete with traditional aerospace industry technology in the hopes that it will prove superior.

I tend to concur with Hertling and Anderson, but with a twist. In fairness, open sources and in general sharing of knowledge and code ARE basic principles that inspire academia. In a sense, this has been going on for a while, and nevertheless progress in creating sentient robots and software have not skyrocketed. Wikipedia, for instance, needed cheap, ubiquitous enabling technologies to manifest itself in all its power.

Sentient robots will need cheap hardware and bodies, making them affordable to everybody. Of course, some good Open Source soul should also gift humankind (and robotkind) with intuitive, cheap, and powerful ways to program these robots.

I will first discuss how we abstracted the behavior of the Create, then discuss the learning rules, followed by a numerical simulation to verify the behavior of the rules we used.

Motor control

The iRobot Create locomotes by means of two monoaxial wheels, whose speeds can be manipulated independently, affording the robot circular and straight trajectories. For simplicity, we limit ourselves to straight trajectories (identical wheel velocities), and rotations about the center of the robot (inverted wheel velocities), and we limit our rotations to multiples of 45° . Thus the robot may move in cardinal or ordinal directions, and we fix our coordinate frame so that these behaviors are oriented with respect to the current target location.

This coordinate frame was chosen for conceptual simplicity. In a fixed or egocentric frame, each behavior must be paired with each possible target orientation in order to characterize it as an “approach” or “avoidance” behavior. In contrast, in a target-centered frame, this extra dimension is reduced, and each behavior need only be considered from one orientation.

Action selection involves mapping a target location to a probability distribution of behaviors, schematized in the following figure, and a selection from that distribution. When an action is selected, a minimal rotation is calculated from the current heading and the target heading, executed, followed by a forward motion until the time segment is ended.

In our implementation, rotations occurred with wheel speeds of 500mm/s (the maximum capability of a Create), with 0.25s allowed per 45° traversed, forward motions used wheel speeds of 100mm/s, and three seconds were allotted to the entire motion sequence. Additionally, for simplicity of calculating turn lengths, target location is given a ±22.5° tolerance, resulting in a minor asymmetry when the target does not align with the cardinal or ordinal directions of the Create.

Learning

With our behaviors characterized, now, as cardinal and ordinal movements with respect to the target, cues must be decided on, to map from a world-state to a behavior. We have established already that a robot topped with a red ball is an aversive stimulus, and a blue ball is appetitive. In addition, it seems useful to characterize the distance of the target as being near or far; an approach behavior is considerably more indicative of an imminent reward or punishment when the target is near than far. Therefore, our cues consist of the pairs (Red, Blue) × (Near, Far), represented by the digits {0..3}. In our implementation, a stimulus is “near” if (and only if) it is < −30° in elevation, relative to the camera. This leads directly to a simple learning matrix L, where L[b,c] encodes the probability of executing behavior b in the presence of cue c. In addition to our 8 behaviors mentioned above, we add an additional stopping behavior, in which the robot orients to the target and attempts to maintain an estimate of target location until the next behavior is selected.

[; \Delta P(b_n|c_n) = \gamma(c_n)e^{(N-n)\Delta t}P(b_n|c_n)\Delta t ;]

Simulation

Training is a time-consuming process, so we produced a reduced simulation which assumes that the robot would move the same distance in each of the cardinal and ordinal directions, and that it can always move in one of those directions with respect to the target. These are not strictly true, but allow us to reduce the problem to one dimension, with each action b(p) representing an update of the current distance to the target p. Table 2 shows the explicit forms of these equations, assuming that each behavior moves a distance x in some direction relative to the target.

Beginning with an initial probability distribution [;P_0 (\cdot|p, c);] dependent on target distance p and color c, let [;p_0;] be the initial position, and suppose we choose our units so that the radius of the robot is 1.

[;b_i \sim P(\cdot|p_i,c);]

[;p_{i+1} = b_i(p_i);]

[;T = \min_t p_t < 1;]

]]> http://www.neurdon.com/2012/03/14/reinforcement-learning-with-trace-conditioning/feed/ 0 http://www.neurdon.com/2012/03/14/reinforcement-learning-with-trace-conditioning/ http://feedproxy.google.com/~r/Neurdon/~3/vsxfXIQSIWk/ http://www.neurdon.com/2012/03/12/learning-approachavoidance-behaviors-for-visual-stimuli/#comments Tue, 13 Mar 2012 01:09:10 +0000 Chris Johnson http://www.neurdon.com/?p=2947 In the fall of 2011, I, along with Jeremy Wurbs and Annan Mozeika, initiated a project to use visual tracking and reinforcement learning to cause an iRobot Create to develop approach and avoidance behaviors. This work was done for credit in Boston University's "Topics in Adaptive Mobile Robotics" course.

Task

The goal of the project was to enable a robotic platform to learn to avoid aversive stimuli and approach appetitive stimuli. A reinforcement learning procedure using trace conditioning was implemented toward this end. A visual tracking system was developed to track red and blue objects, enabling hard-wired behaviors to occur based on the reinforcement learning. Since the training of the real robot was deemed to take tens of thousands of real-time hours, a virtual environment was created to enable the robotic platform to be run, and learn, in simulation. The testing environment included the autonomous robot plus a remote-controlled robot fitted with either a red or blue ball, denoting it as an aversive/appetitive stimulus, respectively. Results are shown qualitatively as learned approaching/avoiding behavior and quantitatively as time between constants under the two different behaviors.

Environment

The physical environment consists of two robots: the autonomous learning robot (shown above as blue with camera), and a controlled target robot (shown above as green with a red ball). The viewing angle for the camera was approximately 60° and the target balls could be detected up to about 2 meters. The autonomous robot was unaware of other obstacles in the environment besides the target (including walls). Thus the environment was bare and the controlled robot was maneuvered to keep the autonomous robot within the environment. When the autonomous robot reached a boundary the trial was reset.

System Overview

This diagram shows the division of the task into stages and the output of each stage. Raw sensory data is fed into both the visual and motor systems. These systems process the incoming data into a set of high level behavioral cues that are sent to higher-order cognitive processing areas. These higher-order areas include target location estimation, reinforcement learning, and select a behavior to implement based on prior learning. The chosen behavior is then sent back to a low-level motor system that implements the desired behavior. The individual systems are described in more detail in further blog posts.

Simulation Overview

As might be expected, performing learning on the robot is not likely to give a wealth of data, particularly for avoidance behaviors, which should result in increased time between learning events, if successfully learned. As a result, to demonstrate the theoretical results, we performed a set of simulations.

Another useful metric is the number of failed trials after each learning trial, shown in the following metric. Here, we see a substantial increase in the number of trials in which the robot failed to contact the target, which confirms that the robot has learned avoidance behavior. (Note that trials are ended at 100 time steps or contact.)

.........

"Q: What's a new project that you've launched in the last year?

A: We have several. One involves cognitive computing, a project we call "Cog ex Machina." That came out of work we are doing with Shell Oil, where we have a tremendous amount of data coming from thousands of sensors in an oil field. Today, human beings look at that data (and interpret it). This project will develop intelligent analytics; it's not quite artificial intelligence but it has similar goals.

Q: Do you think you've missed any good ideas by focusing on a small number of "Big Bets," instead of a larger number of projects?

A: No. We allow people to use 20 percent of their time to look at new areas. In addition, we have seven labs and the director of each lab has 20 percent of the staff working on completely wacky things that I don't know about. At the end of the year, I ask each director, "What happened in your 'Big Bet' projects?" And then, "Surprise me! What else have you done?" The Cog ex Machina project came out of this."

The article, available here, mentions that the authors have "used these chips as a model to study how the connections between neurons strengthen over time, a process thought to be integral to learning and memory, according to a paper published in the Proceedings of the National Academy of Sciences."

Other press coverage is available here. From this article, we learn that "With about 400 transistors, the silicon chip can simulate the activity of a single brain synapse — a connection between two neurons that allows information to flow from one to the other. The researchers anticipate this chip will help neuroscientists learn much more about how the brain works, and could also be used in neural prosthetic devices such as artificial retinas, says Chi-Sang Poon, a principal research scientist in the Harvard-MIT Division of Health Sciences and Technology."

I had two immediate reactions to this work, one good, another not so good.

The first: technically, a very challenging project, and great execution. Much can be learned, in many domains, in trying to bridge the gap between biology and silicon implementations of biological functions. Bravi to the team!

The second reaction: many of the claimed applications of the chip are, to the best of my knowledge, not true. The neuromorphic community has been looking at implementing synapses with a power consumption and dimension orders of magnitude smaller than the ones implemented in this chip. Such an approach (using 400 transistor) for one synapse would not allow any hope for scaling up a system with millions or billions of neurons, and trillions of synapses. Moreover, to be applicable and useful, resulting devices would need to be small. There is no space and power to simulate every aspect of neural computation. The question is: when to stop implementing all details of biological computation. The answer is: When you have the function you need. Do you need to simulate every single aspect of synaptic computation to build a useful application? I do not believe so.

I also believe that the argument that this device would be key in understanding synapses is equally questionable. To approximate a synapse in a digital device, you need to translate analog processes in digital language. This is a (very challenging) exercise in fitting these dynamics in CMOS. What would this tell us on the underlying biology still escapes me, but I leave to the investigators the benefit of the doubt, and I look forward to learn from them what they in turn have learned about biology thanks to this chip. I continue to believe that large-scale software simulations are the primary tools for these sorts of investigations.

Morale: I am sure it has been a challenging task, and much has been learned from the technical standpoint. The research groups should be praised for that, but not for having invented a device that would revolutionize neuromorphic computing, prosthetics, or neuroscience. Of course, I look forward to be contradicted by future publications from this group!

What we were doing was a very early stage in the whole scheme of the research group's project. We were part of this multi-university research group called CELEST (Center of Excellence for Learning in Education, Science, and Technology) that is trying to find out how the human brain works and implement that for real world applications (see BU article Brainy, but so Artificial).

One of the planned future products is a nanoaerial vehicle which is essential a tiny flying vehicle mere centimeters in area. The group foresees this being used in the military where soldiers could throw handfuls of these into unfamiliar territories. The nanoaerial vehicle would be able to fly fully autonomously using optic flow and adapt to the environment without any human intervention. It could map out the entire area and detect where there were people, perhaps even who was a civilian and who was an enemy depending on weapons detection. Its features may not be limited to reconnaissance; depending on how advanced these vehicles become, they could take out targets or act as markers for guided missiles. The possibilities are endless. The group is still far from this goal, but it is a very exciting prospect. What we were doing over the summer was determining whether or not optical flow was a feasible method of navigation for autonomous robots.

Optical flow is the perception of an object's motion due to the object's pixel shifts as the viewer moves relative to the environment. This is how humans and many animals navigate in their environments. Imagine seeing a beach ball while you are at the beach. As you walk closer towards it, it appears to become larger. Also, if the beach ball is not directly in your line of sight, as you approach it, it will appear to move faster on whatever side of the line of sight it originally was from far away. If you are trying to get the ball, as you are approaching it, you will naturally adjust your direction to get the ball directly in your line of sight. For our project, we were trying to apply this navigational technique common in the natural world to autonomous robots.

We used the iRobot Create with a webcam as our platform and ran our optic flow based navigation algorithm in MATLAB. We ran the iRobot Create in a textured environment for optimal optic flow detection.

Our implementation of the optic flow based navigation program successfully traversed through a textured environment (well lit arena with walls covered by randomly generated dots) as displayed in the two videos below. However, as demonstrated by the first video which is running at normal speed, the processing time per frame must be exponentially cut down for optic flow based navigation to become feasible.

Normal Speed

8X Speed

The other main problem we encountered were poor obstacle, wall, and corner avoidance. This was due to the webcam's narrow field of view (55 deg). ( see Obstacle Avoidance Dilemna Diagram)

FUTURE DIRECTIONS:

There are many possible future directions for this project. First and foremost, the biggest issue is the slow image processing time. This is due in part to running the programs through MA TLAB. Implementing the algorithms on a FPGA (Field- Programmable Gate Array) or even ASIC (Application-Specific Integrated Circuit) chip would significantly speed up the computations. Another possible solution would be to write code that would take advantage of GPUs with their parallel processing to do the optic flow computations.

To solve the issues with the narrow field of view, a wide angle webcam could be used. Another possible solution would be to use an array of webcams and then stitch the frames together to form one high quality panoramic image. This would allow the robot to keep obstacles in the field of view and allow the robot to avoid them more proficiently. The same would apply for wall navigation.

Perhaps the most intriguing future direction for this project is to develop a robot q learning system. It would be a points reward system to 'teach' the robot to navigate using optic flow. The robot should be able to eventually avoid obstacles after many trial runs. For example, if the robot is navigating under certain conditions and it runs into an obstacle, it would get a negative point. Whenever the robot makes the correct decision and avoids the collision, the robot gets a positive point. Eventually, after repeatedly committing the same error, it would learn to avoid the obstacle. Ultimately, the robot would learn to adapt on its own. This feature, if successfully developed, would definitely get us closer to having robots that could function and adapt to new environments and situations without human operators.

ABSTRACT

As new technologies continue to develop, more and more robots are replacing humans in situations deemed too dangerous. However, current solutions are not fully automated, requiring offsite human operators for executing basic actions. The ideal solution would be a fully autonomous vehicle that could complete its objectives without any human intervention. In this project, the viability of optical flow based navigation was investigated. Optical flow, or optic flow, is the perception of object motion due to the object’s pixel shifts as the viewer moves relative to the environment. First, motion detection filters were developed and applied to image sequences in MATLAB. Then, they were implemented in optic flow based navigation MATLAB programs for an iRobot Create with a camera to provide video input. The robot successfully traversed through a textured environment but encountered difficulties when attempting to avoid textured obstacles and corners. Experiments were developed to compare the effectiveness of the Correlation and Gabor filters and to find the relationship between increased motion detection ability and processing time per frames. Possible future directions for this project include implementing GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), or even ASIC (Application-Specific Integrated Circuit) chips to speed up computation time, utilizing a wide-angle camera or an array of cameras to get a wider field of view, and integrating a q learning system.

Research Posters

These research posters were presented at the Boston University Research Internship in Science and Engineering Poster Session held on August 12th in the Boston University Life Science and Engineering Department.

Attached is our research paper.

Optical Flow Based Navigation

Attached is our presentation to the Integrated Circuits and Systems Research Group in the Electrical and Computer Engineering Department.

Optic Flow Based Navigation

Let’s begin by considering how the mission of CNS was described at its web site, archived here.  “How does the brain control behavior? How can technology emulate biological intelligence?” These are inspiring questions. Their articulation was visionary and distinctly “ahead of the curve” of recent developments in the ongoing convergence of computation and neuroscience so dear to readers of this blog. Most interestingly, while there are several research labs in the world with cognate mission statements, it is not easy to find comparable language from an academic unit at the grain of a university department with a coherent graduate curriculum. The coordination of talent among the faculty, postdoctoral associates, and graduate students aggregated at CNS was evidently a rare event in academia.

Established as a program offering doctoral and masters degrees in the Fall of 1988, CNS became a department in the 1990-1991 academic year. Its founder and author of the questions that defined its mission statement was Stephen Grossberg, and the only other tenured faculty member at the department’s founding was Gail Carpenter. CNS blossomed by attracting both additional faculty and pioneering graduate students seeking to train in the uncharted interdisciplinary frontiers of neural network modeling. CNS has to date awarded almost 150 PhD degrees, with a few dozen more students in its pipeline. (Although the CNS Department has been decommissioned, its degree-granting program will continue as long as students in good standing progress toward their degrees.) CNS is no longer admitting new students, and prospective applicants are directed to Boston University’s vibrant new  Computational Neuroscience PhD specialization of Boston University’s Graduate Program for Neuroscience.

A lynchpin of the CNS Department’s impact was its unique curriculum of graduate courses, which attracted students from diverse backgrounds, some identifying themselves as neuroscientists and others as modelers or computational scientists. Rather than combining generic neuroscience courses with “straight” mathematics or computing courses (with perhaps examples drawn from neuroscience), the CNS curriculum featured courses that week after week offered tailored expositions of how computational modeling could bridge behavioral and biological data to yield insights on the brain’s ability to control behavior and to point the way to technological emulation of, or prostheses for, the brain’s functions.  These courses covered the domains of vision, learning and memory, audition, reinforcement learning, decision-making, and control of skilled action. The CNS curriculum conferred two important “fringe benefits”: (1) Students learned to construct models across a range of modalities, establishing strong foundations for future research outside their initial areas of specialization. (2) Students helped to cross-train each other and bonded into strong cohorts, because they took not one or two but five, six, or more courses together. Researchers who hired CNS PhDs as postdocs have marveled at how a department whose reputation was built on neural modeling could produce graduates steeped in data, but this was a core outcome of a training program where computational modeling was viewed as a pursuit in service of demanding data, rather than as an activity to be conducted in isolation.

CNS’s approach to research and training was visually summarized by the Outstein logo posted in this article, which is featured on the archival CNS web site and explained in accompanying text, quoted here:

The Outstein was developed by Chris Pribe, CNS ’93, and depicts a combination of the Ehrenstein illusory circle and the outstar neural network design for presynaptic learning developed by Stephen Grossberg. The thickened outer ends of each line represent strength of synaptic connections to neighboring neurons, represented by dots, from a central neuron, represented by the illusory inner disk. The Outstein thus connotes emergent global effects from local processing and the embedding visual illusion is illustrative of the kinds of manifestations of brain processing studied at CNS.

The grand challenge that CNS undertook endures.  Given the level of coordination among researchers that will be needed to unravel the brain’s mysteries, progress requires functional research units of the size of university departments or institutes, rather than efforts headed by individual laboratory heads and their research groups. Research and training in emerging interdisciplinary skills involving mathematics, computation, and the collection and analysis of biological and behavioral data, must be tightly coupled, both administratively and intellectually, through practical training and coursework.

Massively multi-core computers will soon be widely dispersed in academia and industry. As these machines enable the implementation of models with comparable numbers of artificial “neurons” and “synapses” as primate brains, the need for research that bridges neuroscience and technology such as was pioneered by CNS grows ever more urgent. Are facsimiles of the Outstein doomed to wander cached archives of the web forever, like the Flying Dutchman? Increasing numbers of academic units will soon devote themselves to the challenge of fusing engineering with experimental neuroscience through computational modeling; these units will develop intelligent technologies that will both support and help us to understand our human intelligence. The Outstein’s appeal, still real, will not soon fade away.

Unsupervised learning of orientation selectivity maps in a realistic virtual environment
A substantial amount of research has been conducted to show unsupervised learning of oriented receptive field maps from exposure to natural images. In typical scenarios, learning occurs over an extended period of presentation of random patches extracted from a natural image. In this work we sought to test whether receptive fields would develop in an animat wandering in a 3D virtual world. Our neural network consists in roughly three stages: retinal cones, LGN cells and V1 cells. The output from the first stage (retinal cones) was produced by filtering of the rgb image received by an animat with filters whose spectral characteristics correspond to L, M, and S cones. This particular arrangement allowed us to look at learning across multiple achromatic (black, white) and chromatic (red, green, blue, yellow) channels. Note that most of the self-organization work to date has been conducted with a grayscale channel only. For LGN cell center-surround filters, we used self-normalizing distance-dependent shunting equations (Grossberg, 1982), rather than the usual difference-of-Gaussians. For chromatic channels we used cells characterized by dual spatial and cone opponency. For learning we relied on the BCM rule – for Bienenstock, Cooper and Munro, the originators of the rule – with center-surround competition within each cortical hypercolumn.

The animat experienced the virtual environment as shown for example in the movies below.

Example receptive fields learned from various hypercolumns are shown below.

Figure 1. Learned receptive fields. For each channel, receptive fields are grouped per spatial location (many receptive fields are developed at each spatial location). a) Black channel. b) White channel. c) Red channel. d) Green channel. e) Blue channel.

Oriented receptive fields develop in all channels within a few hours of simulation (on a workstation equipped with nVidia GTX-295s GPU). Upon visual inspection it would seem that the receptive fields are not as finely tuned for some of the chromatic channels than for the achromatic ones. This said, more extensive measurements of the tuning properties should be conducted to fully substantiate that claim. In any case, this experiment acts as a good stepping stone toward our next modeling experiment in which we are trying to learn to control saccadic eye movements in addition to learning receptive fields.

For more info, please visit this page.

References
Bienenstock, E.L., Cooper, L. and Munro, P. (1982). Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. The Journal of Neuroscience, 2, 32-48.
Grossberg S. (1982). Why do cells compete? Some examples from visual perception. The UMAP Journal, 3, 103-121.

For the first round, I have built a pretty simple frequency domain classifier for steady-state visually evoked potential (SSVEP) that makes decisions once each four seconds, outputting via UDP to a Python controller script developed by fellow Neurdon contributor and Python guru, Byron Galbraith. SSVEP is a nice choice for this project because the discrete movements are easy to control and the SSVEP signal is robust enough to not worry about false positives on a regular basis. The goal is to get this working in under two second decisions using canonical correlation analysis, however, we are currently using Simulink for the decoder. Anyone wanting to discuss the horridness of Simulink for real-time signal processing will find a friend here. Let's just say the Python IDLE window on my laptop never closes lately.

Here's a short video to give a better idea of what's going on: