galois theory - Construction of non-prime finite fields

I am new to Galois field theory and I am struggling with some definitions. To construct any non-prime finite field $GF(p^n)$ with p prime and $n \in \mathbb{N}$, one has to find an irreducible polynomial $g(x)$ in $GF(p)$ and eventually calculate $GF(p^n) = G(p)[x] / g(x)$.

Assuming I want to construct $GF(9) = GF(3^2)$. Why do I have to do the stuff above? Doesn't $GF(3^2)$ simply contains elements ranging from 0 to 8? What is the upper construction rule about?