Memory is currently (IMHO) the limiting factor of neural networks at general AI stuff. In order to learn new things, they need to "train" by seeing the new thing many times and updating their parameters slowly. Analogous to how humans learn at a low level. You need to practice throwing a ball or riding a bike or speaking a language, and over time you just learn how to do it naturally.
But you can't ask a neural network a question like "what was the last time you saw X". Nor are they very good at storing facts. You can tell it "Obama is the president", but it needs to hear that fact many times before it forms a connection between "Obama" and "president". Whereas a human would remember hearing that fact before and recall it, and eventually after recalling it enough times, would also form that connection at a low level.
This is sort of a step in that direction. They are storing sentences as memories, and then having the NN iterate through all of them and pick the ones that are relevant.
Indeed, another weakness is that multi-layer perceptrons and their deep counterparts are susceptible to "catastrophic forgetting" -- training away old memories when new training data is presented. This is one reason why it's necessary to repeatedly run through the training data. This is a result of the way learning works in these neural networks. Like you say, humans have the ability to learn associatively in one-shot, a very different learning method.
There are neural networks like Grossberg's Adaptive Resonance Theory (ART) which do not suffer from this flaw.
Dropout seems to significantly reduce catastrophic forgetting. So does very sparse activations, supposedly. It's worth noting most NNs are trained with SGD which is an entirely online algorithm.
Not that it's a solved problem but its not unsolvable. Mostly it's that researchers don't care much about online learning since its not too difficult to train offline on stored data.
I could draw an analogy to short-term memory where a new fact is stored separately from the big neural network and it is gradually self-taught to deeper and deeper levels of the network as long as it keeps making sense and fits into the existing set of knowledge. This is much like an allocation sinks deeper in the garbage collector's generations.
Maybe layers of neural networks with different levels of conservation and adaptability would model it better. The first layer just sticks with whatever you give it. If the bit of information seems to work all right, this topmost layer would eventually keep teaching it to a more conservative level below where it takes more time to "learn" and adapt to the new fact. Eventually the information will, in time, reach the "big net" and carve a "memory" for itself there. But the initial trigger was just to push a new fact to the topmost layer, and let it work it out by itself.
That's how human brain works on some naive level. You have the active, conscious mind that can just decide to know or remember something. But that knowledge is brittle: it is easily forgotten unless reinforced several time and the brain can't integrate the new piece of information in its contextual processes that help it generate answers based on deeper analysis, conceptual intercorrelation, and what we call intuition.
The next level is sort of an intuitive insight that you get if you study new information long enough. You just get it and you can start reasoning about it without going through all the steps involved in a strictly conscious way. But it's still like playing with Legos: you know what you're doing but you still need to move the pieces yourself.
The final level is the subconscious where information ends up codified in a way that allows us to go on autopilot. It just happens, you just know things. It's like being able to build a Lego space ship by thinking about one.
For example, let's take driving.
First stage: you're in the driving school and you're overwhelmed by the hundreds of little details every intersection or patch of road offers. Traffic seems chaotic and you only manage to get by if you follow pre-set traffic laws and driving instructions. This is on the conscious level in your active mind.
Second stage: the first year after you received your license. You already have some intuitive grasp on how the traffic works and you think hey this is easy but you still bump into unexpected things once in a while. You might read the lanes and intersections and signs all right but you're surprised by other motorists or events that happens in traffic. This is the level where you've internalized a lot but you're still learning. Many people stop at this level.
Third stage: seasoned driver, typically with several hundred thousand kilometers under his/her belt. Doesn't think about the minutiae of driving but pays attention to "high-level" things such as reading the traffic ahead, identifying cars that might pose a danger of unexpected maneuvers, and adjusting to the average traffic flow and going against the waves of traffic jams instead of echoing and amplifying them.
Another example. I've seen this happen many times eventhough I'm not quite a good mathematician myself. If you read thousands of pages of mathematical papers they first demand a lot of energy to decipher and understand. You'll eventually develop an intuition for how to read the proofs, formulas, and concepts and also for what is expressed between the lines. But the best mathematicians don't just merely think in terms of mathematics but, for some things closer to their area of expertise, they'll just know how everything maps out in relation to the new information.
I've only taken one course in AI, but it looks like everyone is assuming this link's AI is using Artificial Neural Networks. To me it would seem more like an inference engine with a knowledge base. Can anyone find a link that describes what is powering this demo?
It uses a class of RNN they call a MemNN to do something that looks a lot like inference using phrase embeddings.
It's a pretty revolutionary approach - no one seems to have tried anything very similar since the 1990's. The related work section[1] is worth reading.
Interestingly, the Jeff Dean RNN slide deck mentions using RNNs for doing QA, but doesn't expand on what work Google has done in the field. I don't have a URL to hand for that.
They do experiment with loading the MemNN from a set of statements stored as triples, which looks a lot more like a traditional Knowledge Base.
Dunno, might be overlooking lots of complex work, but I'd imagine the more complex work would be taking a book like Lord of the Rings and creating the synopsis that they started with! :)
Not to be too critical... but Frodo isn't in the Shire. He went to the Gray Havens with Bilbo.
“But I have been too deeply hurt, Sam. I tried to save the Shire, and it has been saved, but not for me. It must often be so, Sam, when things are in danger: someone has to give them up, lose them, so that others may keep them.”
https://research.facebook.com/researchers/1543934539189348
http://arxiv.org/abs/1502.05698
http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-1...
So, anyone have open source code that can perform these tasks?