Since Ithaca, Tezos has switched to the Tenderbake consensus algorithm so the accepted answer is no longer valid.
The frozen deposit represents 10% of the highest active stake during the last 7 cycles.
The active stake of a delegate during one cycle is the amount of tez with which it participates in consensus. In general, this corresponds to the delegate’s staking balance, that is, the sum of its own balance and the balance of all accounts delegating to it.
But the active stake can't go over 10 times the delegate’s own balance. Otherwise, there would not be enough funds available for the security deposit. When it does go over, the baker is said to be overdelegated.
The baker may also make it look like its own balance is lower by setting a deposits limit. Then, the active stake can't go over 10 times the deposit limit.
To make calculations for a given cycle, the protocol looks at balances from 6 cycles ago.
Example 1 (most common): a baker has 10k tez in its account and 30k tez delegated to it. The baker didn't set any deposit limit. Its active stake is 40k tez and its security deposit is 4k tez (which is 10% of 40k tez).
Example 2: an overdelegated baker has 10k tez in its account and 190k tez delegated to it. Its active stake is 100k tez (the extra 100k tez are not taken into account). Its security deposit is 10k tez (its entire balance).
Example 3: a baker has 10k tez in its account and 30k tez delegated to it. It has set its deposits limit to 20k tez. Its active sake is 20k tez and its security deposit is 2k tez.
A more rigorous explanation can be found in the Octez documentation.