I have 4 nodes running, all using the same remote signer. In this type of setup, can the baker, endorser and accuser run on either node? Or do they manipulate some local state so they have to group together on the same node?
The endorser and accuser don't need to run on the same system as a node. They can run on a completely separate system, or on the same system (your choice), and only require communication with the Tezos node over RPC (to port 8732 by default).
The baker, however, does require local access to the data files that the node creates. The Tezos blockchain data is stored in an LMDB database, and the baker requires access to these files while it is operating, so the baker should be running on the same system as the Tezos node, and you start the baker by running a command similar to
tezos-baker-003-PsddFKi3 run with local node ~/.tezos-node. This points the baker at the same LMDB database that the Tezos node is using.
While having the baker require a local node is not ideal, the good news is that you can run your endorser on a separate system, or pointing at a separate node, which should allow you to take downtime on your baker (during software upgrades or OS patches) without missing endorsements.