My suggestion would be using tzscan API.
For example checking all activations until you find a match
something like this:
index = 0
loop while true
compare each item
It has a timestamp for each activation
Correct way of doing it with tzscan
I believe the vanilla baker uses the minimal timestamp, from the baking right (as you say, some delay after the previous priority/block): client_baking_forge.ml.
However, you should not rely on this. A baker might choose any timestamp greater than the minimal timestamp. Your node will ignore the block (at the p2p layer) if it is more than 15s in the future.