So since cdk virtualscroll doesn't support multi column, I ended up using ngx-virtual-scroller, which does support multi-columns. The mat-grid-list I also had to let go because of this, but, creating your own tiles isn't that much work when using flexbox.
Here's a snippet using multi columns, [users-online-tile] is a component that represents a user-picture with a name. The IsHandset boolean (from cdk) I use to set the height of the tile so more or less tiles are displayed depending on the screen size.
Hope this helps someone
<virtual-scroller [items]="users" (vsUpdate)="onVsUpdate($event)" (vsEnd)="fetchMore($event)"
(vsChange)="onVsChange($event)" [scrollbarWidth]="20" [scrollbarHeight]="100" [bufferAmount]="100">
<div [ngClass]="{ 'users-online-tile' : (isHandset$ | async), 'users-online-tile-desktop' : !(isHandset$ | async) }"
*ngFor="let user of scrollItems">
<div [users-online-tile]="user" [isHandset]="(isHandset$ | async)"></div>
<!-- <div class="item">{{user.userName}}</div> -->
</div>
</virtual-scroller>