The sex of a baby is determined primarily by the two sex chromosomes. In normal male cells, there's an X and a Y sex chromosome. In normal female cells there are two X chromosomes.
Initially, the embryonic tissue that later becomes testes in boys or ovaries in girls is undifferentiated. At around six to eight weeks of pregnancy, the presence of a Y chromosome causes this tissue to develop into testes. If there's no Y chromosome, ovaries develop.
As well as the hormones released by the testes and ovaries, which determine whether we grow into boys or girls, hormones also influence brain development and behaviour.