The App That Wouldn’t Die: A Workspace ONE Receipt-Collision Bug

If you manage macOS fleets with Workspace ONE UEM, you’ve probably seen it:
an agent gets pushed, installs fine, and then — for no visible reason — gets
uninstalled and reinstalled over and over, forever. Not once. Not twice.
Forever, on a cadence that seems to have no fixed schedule at all.

This is the story of chasing that bug down, the four things that looked
like fixes but weren’t, and the one thing that actually was.

The Symptom

A macOS endpoint kept losing its endpoint management agent. Every check-in
cycle, the Munki log told the same story:

Checking for removals
Removal of Automox Agent added to ManagedInstaller tasks.
...
Removing Automox Agent (1 of 1)...
Running uninstall_script for Automox Agent
Automox agent successfully uninstalled.

And a few minutes (or hours — the cadence was wildly inconsistent) later:

Checking for installs
...
Install of AutomoxAgent-2.6.53.pkg was successful.

Uninstall. Reinstall. Uninstall. Reinstall. All day. The agent’s own event
log showed matching delete/re-add cycles going back at least 24 hours.

First Theory: Server-Side Caching

The first real lead: two different Internal App records in the console
were both installing the exact same underlying package — a current build
and a retired one — and both were assigned (directly or via overlapping
Smart Groups) to the same device. Delete the stale one, problem solved.
Right?

Wrong. Deleting the conflicting app record from the console did nothing.
Neither did:

• Republishing the app
• Sending an MDM sync command via the API
• Clicking “Sync” in the console
• Waiting 45 minutes
• Waiting another hour
• A second, unrelated conflicting app record that was also still
  fighting over the same package (there were two culprits, not one — more
  on that below)

Every one of these should have worked if the problem were simply “the
server hasn’t regenerated the device’s catalog yet.” It kept generating
the exact same removal directive, over and over, no matter how long we
waited or how many syncs we forced.

The Reboot Test

Here’s where it got interesting. A full device reboot forces the Hub
Agent’s launchd jobs to reload from scratch, which — unlike any sync
command — reliably triggers a genuinely fresh manifest and catalog fetch.
If the server had actually fixed its state, a reboot should show it.

It didn’t. Post-reboot, first check-in, same story: fresh catalog fetched,
same stale removal directive, same uninstall, same reinstall a few minutes
later.

That ruled out “just a caching delay.” Something was regenerating this
removal directive from scratch, every single time, independent of what the
console said was assigned.

The Actual Mechanism

Pulling the raw device_catalog.plist off the device (the file Munki
actually consumes) revealed the real problem. Sitting in the array of
pkginfo items was a full entry for the retired app build — the one that
had already been deleted from the console — complete with its own
uninstall script.

And here’s the part that makes this a genuinely nasty bug rather than a
simple stale-record problem: both the current build and the retired
build install the same underlying macOS package receipt.

Old build → receipts: packageid "com.example.agent", version 2.0.28
New build → receipts: packageid "com.example.agent", version 2.6.53

Different pkginfo item names in WS1’s world. Same macOS package identity
in the OS’s world. That’s the collision:

1. Munki installs the new build. macOS registers receipt
   com.example.agent at version 2.6.53.
2. Something is still telling this specific device to remove the old
   item. Its removal condition checks: “is receipt com.example.agent
   present?” Yes — any version satisfies that check. Queue for removal.
3. The old item’s uninstall script isn’t version-aware. It deletes the
   binary and LaunchDaemon plist unconditionally — which happen to be the
   exact same files the new build also uses.
4. The agent is now gone. Next cycle, the new build is missing from
   managed_installs → gets reinstalled → receipt reappears → go to step 2.

An infinite loop, entirely self-inflicted by two catalog entries that
never should have shared a package identity in the first place, and it
will run regardless of what the console’s assignment state says,
because the local device state — not just the console record — was
carrying the stale entry independently.

Why Nothing Server-Side Fixed It

This is the part that cost the most time. The console-visible app
assignment and the actual served catalog are not the same source of
truth in every case. There’s a layer of local device state
(Munki_Repo/manifests, Munki_Repo/catalogs, plus a couple of
WS1-specific plists tracking app status and Munki metadata) that isn’t
guaranteed to get purged just because you deleted the app record upstream.
Once that stale entry is baked into a device’s local files, deleting the
upstream record doesn’t retroactively scrub it — you have to clear it on
the device itself.

The Actual Fix

A short script, run locally with root, that walks four files and deletes
any array entry or dictionary matching the stale app’s package ID or
display name:

• The local Munki metadata cache (munki_data.plist)
• The WS1 agent-status plist (AppStatuses_WS1.plist)
• The local source manifest (device_manifest.plist)
• The local source catalog (device_catalog.plist)

Run it, force one more check-in (a reboot, since sync commands still
weren’t reliable), and the loop actually stopped. managed_uninstalls
went empty and — this was the real test — stayed empty through multiple
subsequent check-ins, no regression.

Interestingly, a second app on the same device hit an almost identical
setup (two catalog entries, same shared package receipt, overlapping Smart
Group assignment) but resolved cleanly with just the console-side
deletion — no local script needed. Same bug shape, two different severity
levels, presumably depending on how long the stale local state had been
sitting there accumulating.

The Takeaways

1. “I deleted it from the console” doesn’t mean it’s gone from the
   device. MDM-managed Munki hybrids keep local state that can outlive
   the record that created it.
2. Sync commands aren’t all equal. An MDM device-sync command updates
   MDM heartbeat/presence; it does not necessarily poke the software
   management agent’s own check-in cycle. If you need a real software
   catalog refresh, a reboot is more reliable than any “sync” button.
3. Two packages that install the same underlying receipt are a loaded
   gun, even if they look like unrelated catalog entries with different
   names and different versions. If your removal logic (or your MDM
   vendor’s) checks receipt presence rather than which catalog item
   installed it, you will eventually get exactly this loop.
4. When something churns forever with no fixed cadence, don’t assume
   it’s a caching delay just because waiting sometimes seems to change
   things. Test it properly — force a genuinely fresh state (a reboot) and
   see if the symptom survives. If it does, the problem is local, not
   server propagation lag.
5. Not every bug needs the same fix. Two apps, same failure shape, two
   different remedies. Diagnose each one on its own before assuming the
   first fix generalizes.

If you’re debugging a Workspace ONE app that seems to be stuck in a
delete/reinstall loop no matter what you do in the console: check whether
two catalog entries share a package receipt, and check the device’s own
local manifest/catalog files, not just what the console says is assigned.