Position of options is important:
ffmpeg [input options] -i input [output options] output
You are trying to apply -c:v libx264
to the input, but you should apply it to the output:
ffmpeg -f gdigrab -framerate 30 -i title="FiveM" -c:v libx264 -f flv rtmp://MYSITE.COM/stream/MYSECRETKEY
With some suggested options added:
ffmpeg -f gdigrab -framerate 30 -i title="FiveM" -c:v libx264 -vf format=yuv420p -g 60 -b:v 3000k -maxrate 3000k -bufsize 6000k -f flv rtmp://MYSITE.COM/stream/MYSECRETKEY
Some services require audio, so you can add silent audio if needed:
ffmpeg -f gdigrab -framerate 30 -i title="FiveM" -re -f lavfi -i anullsrc -c:v libx264 -c:a aac -vf format=yuv420p -g 60 -b:v 3000k -maxrate 3000k -bufsize 6000k -f flv rtmp://MYSITE.COM/stream/MYSECRETKEY