I am not an expert in this case, but here is my somewhat educated opinion and experience.
What is the difference between a Log Group and a Log Stream?
The major functional difference is that the groups “share the same retention, monitoring, and access control settings” while streams don't. So the most obvious case when to split the streams into different log groups is if you want different access control or retention for them, e.g. apps that need to be audited vs nice to have apps.
how they should be used?
However, you can also see the difference in logical grouping, and I believe this is very much up to you. Specifically, I believe you can split the logs of a single service into multiple streams, e.g. error, warn, and info if you are having lots of logs, and don't care about the info when you just want to see the errors.
I am also using the log insights i.e. log search in aws, and I believe they can search only 50 log groups at once so I would most likely group together apps that I am likely to search together often, so that I don't have to manually select each log group every time I am going to search them. I am pretty sure you should be able to have log streams per app and each split into error and info. No clue how worth it is though since you can still just filter logs for “error” string anyway, so most likely just different ways of doing the same thing.
I also believe that the default way of collecting logs for kubernetes and its apps is that it is a single log group for the whole cluster, and each container per pod has its own log stream. I am not sure this is good though because you cannot be granular on your access control and retention if different pods have different security and audit requirements. However, I am sure it is possible to define log groups per different pods, I just never had the need to split them.
As for Lambdas and ECS, sometimes we have a log group just for a single lambda, while others are grouped if they are working closely together. ECS is usually already split into logical groups using clusters so log group per ECS cluster.
it seems like I would want to create a new Log Stream for each machine or each user per machine. Does this sound right?
You certainly should be able to do it, although I would personally go for a log group, with streams per each thematically connected app, and usually the logs that are associated with some user would have a user ID or some tracking ID if you have some orders, and I would filter the logs based on that. Overall though, I would claim though that it is more of a personal preference than anything else.
TL; DR I think it is just personal taste on logical groupings if access control and retention doesn't matter.
