2009-11-20

Why pigs will fly (or the curious case of UPHClean and the SYSTEM profile)

My first response to finding domain Group Policy user settings in the SYSTEM profile on our terminal servers was "and pigs will fly" - i.e. not bl**** likely!

Still, the fact remained: restrictive domain policies in the domain were creeping into the SYSTEM profile (HKEY_USERS\S-1-5-18 -- for which incidentally HKEY_USERS\.DEFAULT is an alias). In most cases this wouldn't be a problem, but for us it was, since SCCM machine installation packages were mysteriously failing on the affected servers, and on closer inspection it turned out that the local system account was being denied running install packages! Hmm... strange ... we're talking about localsystem here! Digging into the registry we found the key and value  HKEY_USERS\S-1-5-18\Software\Microsoft\Windows\CurrentVersion \Policies\Explorer\RestrictRun that limits perimitted applications to those listed under the RestrictRun key. The application list was easily recognizable as the same one set in the domain user Group Policy "Administrative templates/System/Run only allowed Windows applications" - and does not include msiexec.exe.

This is all fine and well when applied to users John and Jane logging onto the terminal servers, but not for the local system account! So how was this user setting creeping into the SYSTEM profile? Even when we removed the setting, it reappeared within a week or so! Be assured: all likely and not so likely explanations were considered, and I'll spare you the boring details. More or less on a hunch, I disabled UPHClean (v. 1.6.30.0) on the affected servers: 

The User Profile Hive Cleanup service helps to ensure user sessions are completely terminated when a user logs off. System processes and applications occasionally maintain connections to registry keys in the user profile after a user logs off. In those cases the user session is prevented from completely ending. 
This seemed to stop the problem from reappearing - but what was going on? Completely disabling UPHClean was also not an attractive long term solution, so we had to dig deeper. The following from the readme file is a hint:
UPHClean assists the operating system to unload user profile hive by remapping the handles to the user profile hive to the default user hive. For example if a process has a handle to HKEY_USERS\S-1-5-21-X-Y-Z\Software\Microsoft after remapping it would have a handle to HKEY_USERS\.DEFAULT\Software\Microsoft.  This allows the profile hive to unload.
Hmmm... what if the process owning the remapped handle decides to write data to the user hive, and what if that data happened to be policy values? Seems awfully close to what's going on. The misbehaving servers were monitored to find out exactly when the settings turned up in the system profile. UPHClean logs application log event ID 1401 when it remaps a handle and interestingly there are in fact some hits on event 1401 mentioning various "Policies" subkeys in a time frame of maximum 60 minutes before the system profile received the value in all observed cases. 

The following test nailed it:

  1. Log on user "A" to a machine running UPHClean.
  2. Log on user "B" in a new session on the same machine.
  3. Connect to the currently loaded user hive for user "A" with regedit.
  4. Create a key and a value under "Software", for example key name "Foo", Value name "Bar" (REG_SZ) = "Test value". This is now stored in users A's hive.
  5. Double-click "Bar" and modify the contents to "New test value" (but do *not* close the "Edit string" dialog yet)  
  6. Log off user "A", and check the app event log. You should now see UPHClean event id 1401.
  7. Now close the "Edit string" dialog opened in step (5).
  8. Check under HKEY_USERS\S-1-5-18\Software: Magically, you'll now find the value HKEY_USERS\S-1-5-18\Software\Foo\Bar = "New test value" in that hive!
The fix
Fortunately UPHClean can be tweaked to change the default behaviour. In our case we decided to disable handle remapping entirely by setting the config value REMAP_HANDLE_PROCESS_LIST to a single dash ("-"). By default this setting is an asterix ("*") interpreted as "remap for all processes". The full path to the setting is:
HKLM\SYSTEM\CurrentControlSet\Services\UPHClean\Parameters REMAP_HANDLE_PROCESS_LIST
It would be nice to be able to configure this using a custom admin template and Group Policy, however, it is a REG_MULTI_SZ value, so you need to script it one way or another.

I'm told UPHClean 2.0 and the "User Profile service" in Windows Server 2008 implement handle remapping a bit differently, so hopefully this will not be a problem with those versions. But with UPHClean 1.6.30 in its default configuration you can get user Group Policy settings in the localsystem profile  - and pigs will fly.

No comments: