A faster way to retrieve all the VM's associated to a portgroup

I have a script that I use to add/remove portgroup to a large set of VM's. I'm curious if there's a faster to get the list of VM's associated to a port group.

Right now, a command like this

Get-VM |Get-NetworkAdapter | Where {$_.NetworkName -eq "PortGroupName" } 

Takes a long time as it is running for each VM.

When I log into VSphere, I see all the VM's under a portgroup right away, is there a better way through PowerCLI?