Python NEAT not learning further after a certain point

It seems that my program is trying to learn until a certain point, and then it's satisfied and stops improving and changing at all. With my testing it usually goes to a value of -5 at most, and then it remains there no matter how long I keep it running. The result set does not change either.

Just to keep track of it I made my own kind of logging thing to see which did best. The array of ones and zeroes is referring to how often the AI made a right choice (1), and how often the AI made a wrong choice (0).

My goal is to get the AI to repeat a pattern of going above 0.5 and then going below 0.5, not necessarily find the odd number. This was meant as just a little test to see if I could get an AI working properly with some basic data, before doing something a bit more advanced.

But unfortunately it's not working and I am not certain why.

The code:

import os
import neat

def main(genomes, config):
    networks = []
    ge = []
    choices = []

    for _, genome in genomes:
        network = neat.nn.FeedForwardNetwork.create(genome, config)
        networks.append(network)

        genome.fitness = 0
        ge.append(genome)

        choices.append([])

    for x in range(25):
        for i, genome in enumerate(ge):
            output = networks[i].activate([x])

            # print(str(x) + " - " + str(i) + " chose " + str(output[0]))
            if output[0] > 0.5:
                if x % 2 == 0:
                    ge[i].fitness += 1
                    choices[i].append(1)
                else:
                    ge[i].fitness -= 5
                    choices[i].append(0)
            else:
                if not x % 2 == 0:
                    ge[i].fitness += 1
                    choices[i].append(1)
                else:
                    ge[i].fitness -= 5
                    choices[i].append(0)
                    pass
            
            # Optional death function, if I use this there are no winners at any point.
            # if ge[i].fitness <= 20:
            #     ge[i].fitness -= 100
            #     ge.pop(i)
            #     choices.pop(i)
            #    networks.pop(i)
    if len(ge) > 0:
        fittest = -1
        fitness = -999999
        for i, genome in enumerate(ge):
            if ge[i].fitness > fitness:
                fittest = i
                fitness = ge[i].fitness

        print("Best: " + str(fittest) + " with fitness " + str(fitness))
        print(str(choices[fittest]))
    else:
        print("Done with no best.")

def run(config_path):
    config = neat.config.Config(neat.DefaultGenome, neat.DefaultReproduction, neat.DefaultSpeciesSet,
                                neat.DefaultStagnation, config_path)

    pop = neat.Population(config)

    #pop.add_reporter(neat.StdOutReporter(True))
    #stats = neat.StatisticsReporter()
    #pop.add_reporter(stats)

    winner = pop.run(main, 100)

if __name__ == "__main__":
    local_dir = os.path.dirname(__file__)
    config_path = os.path.join(local_dir, "config-feedforward.txt")
    run(config_path)

The NEAT config:

[NEAT]
fitness_criterion     = max
fitness_threshold     = 100000
pop_size              = 5000
reset_on_extinction   = False

[DefaultGenome]
# node activation options
activation_default      = tanh
activation_mutate_rate  = 0.0
activation_options      = tanh

# node aggregation options
aggregation_default     = sum
aggregation_mutate_rate = 0.0
aggregation_options     = sum

# node bias options
bias_init_mean          = 0.0
bias_init_stdev         = 1.0
bias_max_value          = 30.0
bias_min_value          = -30.0
bias_mutate_power       = 0.5
bias_mutate_rate        = 0.7
bias_replace_rate       = 0.1

# genome compatibility options
compatibility_disjoint_coefficient = 1.0
compatibility_weight_coefficient   = 0.5

# connection add/remove rates
conn_add_prob           = 0.5
conn_delete_prob        = 0.5

# connection enable options
enabled_default         = True
enabled_mutate_rate     = 0.1

feed_forward            = True
initial_connection      = full

# node add/remove rates
node_add_prob           = 0.2
node_delete_prob        = 0.2

# network parameters
num_hidden              = 0
num_inputs              = 1
num_outputs             = 1

# node response options
response_init_mean      = 1.0
response_init_stdev     = 0.0
response_max_value      = 30.0
response_min_value      = -30.0
response_mutate_power   = 0.0
response_mutate_rate    = 0.0
response_replace_rate   = 0.0

# connection weight options
weight_init_mean        = 0.0
weight_init_stdev       = 1.0
weight_max_value        = 30
weight_min_value        = -30
weight_mutate_power     = 0.5
weight_mutate_rate      = 0.8
weight_replace_rate     = 0.1

[DefaultSpeciesSet]
compatibility_threshold = 3.0

[DefaultStagnation]
species_fitness_func = max
max_stagnation       = 20
species_elitism      = 2

[DefaultReproduction]
elitism            = 2
survival_threshold = 0.2

Sorry to tell you that this approach just isn't going to work. Remember that neural networks are typically built out of doing a matrix multiply and then max with 0 (this is called RELU), so basically linear at each layer with a cutoff (and no, picking a different activation like sigmoid is not going to help). You want the network to produce >.5, <.5, >.5, <.5, ... 25 times. Imagine what it would take to build that out of RELU pieces. You would at the very least need a network that is ~25 layers deep, and NEAT is just not going to produce a network that large without consistent incremental progress in the evolution. You are in good company though, what you are doing is equivalent to learning the modulo operator, which has been studied for many years. Here is one post that succeeds, though not using NEAT. Keras: Making a neural network to find a number's modulus

The only real progress you could make with NEAT is to give the network many more features as inputs, e.g. give it x%2 as input and it will learn quickly, though this is obviously 'cheating'.

Recommended topics

Hot tags