`That` is restrictive:
This implies that we have more than one office, but the office that is has a gym (the others do not), is in London. The `that` in that sentence is essential, without it, the meaning of the sentence would be lost.
Our office that has a gym is in London.
`Which` is non-restrictive, and adds non-essential information, when removed, the sentence would still make sense. For instance:
This means that we have only a single office, and the `which`-clause is merely adding extra information that is has a gym. It could be re-written as two sentences:
Our office, which has a gym, is in London.
Our office is in London. It has a gym.