The client IP address in the SSH connection will be part of the range 35.235.240.0/20. This range is the pool of IP addresses used by IAP to proxy the connection from your browser to your instance. So, you can create a more restrictive VPC firewall rule allowing SSH connections only from this IP address range. As a result, only users allowed by IAP will be able to connect to VM using SSH.