Remove Permission Weirdness

I have the following script to create a folder and add a default permission.

It is then supposed to remove inheritance.

Then remove Authenticated Users from the folder.

If I run the following script all works except remove the Authenticated Users from the folder.

$dir = "\\serversfs01\users\systest3"

New-item -Path $dir -ItemType Directory

$acl = get-acl -Path $dir
$rule = New-Object System.Security.AccessControl.FileSystemAccessRule('systest3',"DeleteSubdirectoriesAndFiles, Write, ReadAndExecute, Synchronize", "ContainerInherit, ObjectInherit", "None", "Allow")
$acl.AddAccessRule($rule)

# This removes inheritance
$acl.SetAccessRuleProtection($True,$True)

$removeuser = "NT Authority\Authenticated Users"
$accessrule = New-Object system.security.AccessControl.FileSystemAccessRule($removeuser,"Read",,,"Allow")
$acl.RemoveAccessRuleAll($accessrule)

$acl |Set-Acl

I can then run this and it removes the Authenticated Users from the folder. I just can't figure out why it does not work all as one.

$dir = "\\serversfs01\users\systest3"

$acl = get-acl -Path $dir

$removeuser = "NT Authority\Authenticated Users"
$accessrule = New-Object system.security.AccessControl.FileSystemAccessRule($removeuser,"Read",,,"Allow")
$acl.RemoveAccessRuleAll($accessrule)

$acl |Set-Acl

Any ideas. I thought it was a timing issue, but that is not it.