Unitary operator

Introduction

They are a generalization of unitary matrix.

A unitary operator is a bounded linear operator U:HH on a Hilbert space H that preserves the inner product. In other words, for any vectors x and y in H, we have:

Ux,Uy=x,y

Another equivalent definition is that an operator U is unitary if it is invertible and U=U1, where U is the adjoint of U.

They have properties like preserving norms (Ux=x) and being "distance-preserving" in the Hilbert space.

Relation to self adjoint operators

They are in close relationship to self adjoint operators, given by the Stone's theorem. Roughly speaking, every 1-parameter subgroup of unitary operators is generated by a self adjoint operator (multiplied by i).
Two unitary operators commute if and only if their self adjoint operators commute: if two unitary operators U1 and U2 commute, meaning U1U2=U2U1, then their generating self-adjoint operators A1 and A2 will also commute, A1A2=A2A1.
Let's consider that the unitary operators are generated by self-adjoint operators as:

U1(t)=exp(itA1),U2(t)=exp(itA2).

If A1 and A2 commute then

exp(itA1)exp(itA2)=exp(itA1+itA2),

and

exp(itA2)exp(itA1)=exp(itA1+itA2),

so U1(t),U2(t) commute.

In Quantum Mechanics

They play a fundamental role in quantum mechanics, where they often represent evolution though a parameter: time evolution, continuous rotations, and so on...
Indeed, they are the biggest set of accepted transformations of a quantum mechanical system (any other transformation would "break the stage" in which everything is happening, the corresponding Hilbert space). If we additionally require to the unitary transformations to preserve the specific Hamiltonian of the system then we obtain the symmetry group of a physical system.