How to update weights in keras for reinforcement learning?

Question

I am working in a reinforcement learning program and I am using this article as the reference. I am using python with keras(theano) for creating neural network and the pseudo code I am using for this program is

Do a feedforward pass for the current state s to get predicted Q-values for all actions.

Do a feedforward pass for the next state s’ and calculate maximum overall network outputs max a’ Q(s’, a’).

Set Q-value target for action to r + γmax a’ Q(s’, a’) (use the max calculated in step 2). For all other actions, set the Q-value target to the same as originally returned from step 1, making the error 0 for those outputs.

Update the weights using backpropagation.

The loss function equation here is this

where my reward is +1, maxQ(s',a') =0.8375 and Q(s,a)=0.6892

My L would be 1/2*(1+0.8375-0.6892)^2=0.659296445

Now how should I update my model neural network weights using the above loss function value if my model structure is this

model = Sequential()
model.add(Dense(150, input_dim=150))
model.add(Dense(10))
model.add(Dense(1,activation='sigmoid'))
model.compile(loss='mse', optimizer='adam')

Unwelcome

Answer 1

Assuming the NN is modeling the Q value function, you would just pass the target to the network. e.g.

model.train_on_batch(state_action_vector, target)

Where state_action_vector is some preprocessed vector representing the state-action input to your network. Since your network is using an MSE loss function, it will compute the prediction term using the state-action on the forward pass and then update the weights according to your target.

Concrescence

Please provide more detailed description. Thanks – Vondavonni