 |
|
|
 |
|
|
|
|
Swarvy |
| | | | | | posted 8/31/2010 16:39 |   | | | | | | | | | Hello, I'm getting started on writing my own Neural Network code, and although I've only written the input/output system at the moment so that the data can be fed into it etc, and haven't started on the ANN itself, I've been doing quite a lot of reading about neural networks and have a few questions.
Question 1 - Can you use a neural network to produce a non-binary output or should the output of an ANN always either be 1 or 0? If so, thats a little limited for most situation, isn't it?
Question 2 - From the reading I've been doing it seems that often only one hidden layer is implemented in most ANNs, I was wondering why more hidden layers aren't implemented in most cases?
Question 3 - I understand the point of the training sets and validation sets, but how is the validation set any different from the generalization set when it comes to learning? Surely if you input the training set to calculate the neural network weightings, and then use the validation set to test the network's accuracy, what does the generalization set do?
Question 4 - Since the performance of a neural network at solving a given problem depends quite substantially on the structure of the network chosen, could it be possible to get the program to find its own optimum network structure. I was thinking of implementing a system where I have a big population of similar neural networks (probably 2000 to 5000 ANNs) and then after each network has learned, taking an average of their accuracies and then allow random adjustment to the structure within a given framework (e.g. input and output layer must remain the same structure, but additional neurons or hidden layers could be added in the middle) by using an evolutionary-based algorithm. I.e. The neural networks with the best accuracy carry on their traits to the next generation etc. Would that be possible? Would that be a good idea, so that the neural network self-improves? I am concerned that if I did that it would carry on adding neurons and would never find an optimized structure, in which case I would have to add in additional stopping conditions such as maximum network size, an accuracy threshold etc.
Question 5 - How many outputs can an ANN give? In all the examples I've seen they only ever give out a single binary value. Ideally I want to get 4 or 5 outputs from my network, how feasible is that as a suggestion?
Question 6 - Would it be possible to do the idea suggested in question 4, but instead using it to optimize the momentum based on the data. Add in random variation and see which ones perform best?
Needless to say if I decide to include all that evolution stuff my program is gonna be a bit of a monster to program, especially since I've never done any AI programming before, but I'm pretty good at C/C++ so should be able to cope.
| | |
| | |
AiHasBeenSolved |
| | | | | | posted 8/31/2010 17:25 | | | | | | | | | | Neural Networks may be the wrong way to go, if you seriously want to do AI. You might consider porting the pre-existing MindForth AI into C++ and learn about the simulated neural nets involved as you go.
| | |  http://mind.sourceforge.net/aisteps.html | | |
| | |
tkorrovi |
| | | | | | posted 8/31/2010 21:11 | | | | | | | | | | I wouldn't say that neural network is a wrong way to go. Neural network is good for certain tasks, but only for certain tasks. So everything depends on what one wants to do.
| | |
| | |
Rexmyshrads |
| | | | | | posted 9/13/2010 07:52 | | | | | | | | | | I'm not an expert so don't mind me if I sound like it, but if you look at pretty much anything alive it needs sleep. Why? Well sleep is when your brain does this organizing or should I say consolidation of things it has learned. The structure of your memories have already been prenetworked but the consolidation comes when you enter REM sleep. Your Brain actually fries itself thereby reducing the number of connections to any given pathway or deleting them all together. Maybe an aspect of the AI program you could focus on is thedistribution of multi evolutionary paths.Try something like a distributed network solution and harness the internet. if you could get the simulation done on many different computers at the same time then get many different other computers to do the weighting and pass on the consolidated results to you it would relieve a lot of the burden upon you and you could focus instead on the actual organizational AI
| | |
| | |
tkorrovi |
| | | | | | posted 9/13/2010 09:08 | | | | | | | | | | Yes right, when there are no external stimulants, then the brain works more internally. And deleting the old connections is almost as important as creating new. In my system they can be deleted, or they decay by themselves, but this again needs a dynamic system, and this is not a neural network.
| | |
| | |
simnia |
| | | | | | posted 8/16/2012 08:08 | | | | | | | | | | The post above that said neural networks are the wrong way to go is pretty much correct. Read or browse Simon Haykin's book "Neural Networks: A Comprehensive Foundation", which after several hundred pages of deep mathematical analysis concludes that neural networks are little more than a rewritten version of existing statistical methods, except for their analog activation functions, which may not even contribute any new functionality that is useful. I heard an expert in neural networks say almost the same thing at a conference before Haykin's book even existed.
Question 1 - Can you use a neural network to produce a non-binary output or should the output of an ANN always either be 1 or 0?
Non-binary output is fine. The math that describes the output of each unit is all in real numbers that represent a continuum of voltages, which you can use directly.
Question 2 - From the reading I've been doing it seems that often only one hidden layer is implemented in most ANNs, I was wondering why more hidden layers aren't implemented in most cases?
Either they're not needed for the desired mapping, and/or they take too long to train, and/or you're reading pedagogical material, and/or you're reading about preliminary experiments. Plenty of companies and researchers using ANNs for serious projects use more layers.
Question 3 - I understand the point of the training sets and validation sets, but how is the validation set any different from the generalization set when it comes to learning?
Sorry, I'm not familiar enough with those topics to answer.
Question 4 - Since the performance of a neural network at solving a given problem depends quite substantially on the structure of the network chosen, could it be possible to get the program to find its own optimum network structure.
I know of a couple people who have tried variations of this idea. If I had to summarize why this isn't a very promising idea, I'd have to say: (1) none of those people's ideas went anywhere; (2) you're trying to get dumb networks to teach other dumb networks, so it's a case of the blind leading the blind; (3) it sort of reduces to the question of "Where do computer programs come from?" since you'd be trying to get human capability materialize out of nowhere. It's true that nature did that, but nature also had millennia in which to work on the problem, which you don't have.
Question 5 - How many outputs can an ANN give?
Unlimited.
Question 6 - Would it be possible to do the idea suggested in question 4, but instead using it to optimize the momentum based on the data. Add in random variation and see which ones perform best?
Same answer as suggested for Question 4.
| | | Last edited by simnia @ 8/18/2012 4:51:00 AM | | |
| | |
tkorrovi |
| | | | | | posted 8/16/2012 17:23 | | | | | | | | | | What i have always said about artificial neural networks is that they cannot model the processes in their environment. This makes them inherently restricted, so it is unlikely that any improvements help.
| | | Artificial Consciousness ADS-AC project | | |
| | |
simnia |
| | | | | | posted 8/17/2012 01:11 | | | | | | | | | | Excellent insight, tkorrovi, as was another comment of yours from another thread about the limitations of symbolic systems.
Artificial neural networks and expert systems are clearly old technology that doesn't seem to have the potential to go anywhere in the long run, which is also my opinion of Case Based Reasoning systems, Genetic Algorithms, Semantic Web, data mining, Bayesian Nets, Non-monotonic reasoning, probably also Soar, and probably every other main idea that has been published so far. I've seen at least one post in this forum asking about the state of A.I. and where it's going, and what's most promising, but from what I've seen and heard inside and outside of this forum, there is some interest in agents and chatbots, but the overwhelming impression I'm getting of A.I. now is that it's in an uncomfortable holding pattern, waiting for some major new development, and nobody wants to pay people to think about or to develop such ideas, especially with the world economy in shambles, especially in the United States.
For that reason, I think this forum is a good place to hang out. At least four people here, including myself, are clearly working on their own ideas that they claim are more promising than anything anybody else is working on. I believe it is from these relatively unknown people working on really unique ideas without pay, people who are also aware of the lamentable state-of-the-art in A.I. and why the previous ideas have failed, that the most promising future ideas are likely to develop. So stick with it, folks. Keep digging, because the treasure is there, somewhere. The field is looking for somebody to lead the way, so there are terrific opportunities in the field for those who keep the faith.
I do have one idea related to putting more intelligence into artificial neural networks that I've never seen anyone tackle, an idea that would almost definitely push the state of the art somewhat, in a direction similar to what Swarvy is proposing, but ultimately I still think neural networks are a dead end so I personally don't want to spend time on it. So here's the idea for free if anybody is looking for a novel project related to neural networks...
Kohonen maps are painful to watch as they develop because often there is an obvious geometrical pattern (like a rectangular grid, or curved rectangular grid) toward which the network is tending, but the network has no global detection or assessment ability to speed up the convergence to that obvious upcoming pattern.
http://en.wikipedia.org/wiki/Self-organizing_map
My idea is that if such awareness could be programmed into the network, that would greatly speed convergence. This idea would also be applicable to other types of neural networks that do mappings, such as backpropagation networks, so success here would benefit the whole field. A concept called Kolmogorov complexity provides just such a measure.
http://en.wikipedia.org/wiki/Kolmogorov_complexity
For example, a global/textual assessment such as that the system is tending to a tiled pattern "with 4-sided tiles, of length:width ratio of 3:1, orientated at angle 90 degrees" would provide a simple, abstract description consisting of only three attributes containing a total of three parameters that would describe an entire tile pattern. This is like a "mini-theorem", like an equation in physics, that tries to find the simplest description that provides the most accurate description of system behavior. (= Occam's Razor). As soon as the Kohonen net detected a similarity of its developing pattern to such a description, it could immediately adjust its weights to jump right to that tile pattern as the most likely outcome. It wouldn't become permanently hung up on that mini-theory, though, since if another pattern with less Kolmogorov complexity arose, it would then jump to that prediction instead.
My impression of Kolmogorov complexity is that it's a general concept, not very well-defined in detail, and that implementation of it requires the user to come up with their own metrics, but that's no problem.
Anyway, that's my donated idea if anybody wants to run with it and to produce some technical articles with appealing drawings. The foundation of that idea, Kolmogorov complexity, was used as the foundation of that Swedish program I mentioned in another thread that solves IQ test problems, so that general approach is promising and can be applied to other areas. To me, that would be the real value of pursuing this neural network enhancement idea, not so much for the value for neural networks per se.
| | |
| | |
tkorrovi |
| | | | | | posted 8/17/2012 05:38 | | | | | | | | | | | |
simnia wrote @ 8/17/2012 1:11:00 AM:
For example, a global/textual assessment such as that the system is tending to a tiled pattern "with 4-sided tiles, of length:width ratio of 3:1, orientated at angle 90 degrees" would provide a simple, abstract description consisting of only three attributes containing a total of three parameters that would describe an entire tile pattern.
| |
So, generating an algorithm which creates a pattern in accordance with that model with the minimal Kolmogorov complexity which matches the original pattern, enables to reduce the Kohonen map? The problem with that is that we need some model or set of models based on which we generate such algorithms, these models are pre-defined and that restricts the system.
But i think your idea is good, enables to improve neural networks and may provide very good results for some applications. Definitely it makes sense to do a research in that direction, this if anything in the research of neural networks, makes sense.
I cannot be sure though that i completely understood your idea. There are things which i don't understand, and i likely fail the criterion the way some understand smartness. I don't understand everything at once, i may also know two things which are associated, and it may take months until my brain brings these two things together. This certainly does not correspond to everyone's idea of what smartness is, and what innovative thinking is.
My theory (below) is the derived solution, proceeding only from the assumption that AI must be unrestricted in self-development (approaching unrestrictedness). So different from other approaches it is not just "maybe there is the solution", it is "the solution must be there". This needs research, but i'm not working on my theory now because there is no interest, that is, no one is interested in participating in that research. That's the same with your idea i think, you put it up here so that someone else would develop it further. Unfortunately no one comes here who is capable and willing to do that research.
I like you in that you constructively deal with arguments and points of view which contradict your points of view. A quality which i find quite rare in all people who i have talked with, having different academic qualifications or not having. I really appreciate that, and i would say out of many, you are a one of these few people with whom it makes sense to work together with.
In fact, there should be separate threads for discussing further your idea, my idea, and perhaps also someone else ideas, by these who are interested. As talking further about these things here becomes off topic, though i think to some extent it is ok, as this forum always has been quite a mess.
| | | Artificial Consciousness ADS-AC project | | | Last edited by tkorrovi @ 8/17/2012 6:10:00 AM | | |
| | |
simnia |
| | | | | | posted 8/18/2012 06:20 | | | | | | | | | | | |
tkorrovi wrote @ 8/17/2012 5:38:00 AM:
So, generating an algorithm which creates a pattern in accordance with that model with the minimal Kolmogorov complexity which matches the original pattern, enables to reduce the Kohonen map? The problem with that is that we need some model or set of models based on which we generate such algorithms, these models are pre-defined and that restricts the system.
| |
It wouldn't reduce the map: it would speed up convergence to the predicted map. Convergence speed is always an issue in ANNs, so I believe this would be of general interest in the field. In some sense such a feature might not even help: what it would essentially do is substitute a long, slow convergence process with a fast convergence process where nothing would be seen to change for long periods of time, which might be even more frustrating! But my proposed enhancement could be added as another system on top of the net so that both processes could be viewed at the same time, and if there *is* a simple map, this proposed method would probably find it in a hurry.
You are exactly right: To implement my idea you'd have to create a predefined set of features/descriptors as a library that the network could use, so that finite library would restrict the system. That's one of the reasons I'm not keen on spending time on this particular idea. In defense of my idea, though, I believe the nature of the problem is such that the limitation you mention would be so minimal that it would be almost negligible in relation to the benefit gained.
I can post more details about this ANN idea tomorrow in a separate thread when I have more time, though I'm reluctant to even get that far into it.
My wording may have been misleading in my previous post: my invention that I'm avidly pursuing at the moment is different than this Kohonen net idea I threw out for free. Until I get some publications out that establish my precedence on my other invention, I'm not going to discuss my invention in detail anywhere. That's good for everyone, anyway, since I could just be mistaken and might eventually have to admit that I have nothing novel of value to offer the field of A.I., now or ever. )-: But thanks for the compliments, anyway!
| | | Last edited by simnia @ 8/18/2012 8:19:00 PM | | |
| | |
tkorrovi |
| | | | | | posted 8/18/2012 20:41 | | | | | | | | | | | |
simnia wrote @ 8/17/2012 1:11:00 AM:
At least four people here, including myself, are clearly working on their own ideas that they claim are more promising than anything anybody else is working on. I believe it is from these relatively unknown people working on really unique ideas without pay, people who are also aware of the lamentable state-of-the-art in A.I. and why the previous ideas have failed, that the most promising future ideas are likely to develop. So stick with it, folks. Keep digging, because the treasure is there, somewhere.
| |
At least thank you for that.
But this development is stooled. At least these four people should then stick together for there to be any hope. If someone attacks anyone of these, they are alone against the whole world. People have written to me that they like my theory, by email, just to not say that publicly. All that appreciation is worth nothing because i cannot show to anyone that any soul in the world takes me seriously. Someone says, nice you are in this forum, thinks a worthless person like me should be full of joy when hearing this. And that after changes this opinion and thinks it's better if i were not in this forum. Developing these ideas takes many years, it's a very hard work, they think award for that is saying, ok you are not so disgusting that i don't want to talk with you at all. But i am Irish, you know, so you are anyway nothing compared to me, and you should consider it a great honor if i, Irish, ever agree to talk with you. With such attitude, which is common in the Internet, everything falls into the common ignorance, where everything goes. In science they have reviews, but here there is not a single person who says anything positive about anyone's else idea. You say the solution is the alternative, but how can such way there be any alternative?
> The field is looking for somebody to lead the way, so there are terrific opportunities in the field for those who keep the faith.
I doubt that. I think that the field is defined, somewhere along the behaviorist / transhumanist / applied science lines, to serve some agenda, and the real solution is beyond doubt outside it, so if anyone finds it, this is the person they need the least. And as i have said, what concerns True AI, this is only about research, there cannot be any immediate practical benefits, so one cannot make any profit out of it. Btw, what concerns profit, the today's business world is anyway controlled by a few big corporations, so what makes profit today is what is in the interest of these corporations, no more are the real practical benefits what matter. And if the real solution is not in their desired scope, then they most likely reject any such idea.
| | | Artificial Consciousness ADS-AC project | | | Last edited by tkorrovi @ 8/18/2012 8:53:00 PM | | |
|
|
|
|
|
