Matthew 2:1 indicates that the wise men came sometime after the birth of Jesus. It was not during His birth, as is customary to the nativity scenes that we are familar with. The event of the wise men more than likely took place after Jesus was dedicated to God in the temple. I’d say Jesus was about 2 years old when the wise men presented Him with gifts. In that case, Jesus was a todler not an infant. What is your view?
Note also that Matthew states the Magi visited them in "a house", not in a stable. It does seem to indicate that following their trip from Nazareth for their taxation, that they remained in Bethlehem. That was Joseph’s home town and his family was there. as Nazareth was a tiny place with only a couple hundred homes where their major industry was funerals (most of the "town" was graveyards), Joseph probably felt he had a better chance finding work and supporting his family in Bethlehem.
The idea of "2 years" came from Herod’s order to kill all male children up to that age. There is evidence to show the Quirinius was governor over Syria for his first term between 6 and 5 BC, and that Herod died in 4 BC. So Jesus could have been born as early as 6BC, lived in Bethlehem for a time, and the gone to Egypt until 4BC. How long he was in Egypt is also not given.
It appears the oldest that Jesus could have been when the Magi visited was 2. Born in 6 BC and fled in 4 BC to only be in Egypt for a few weeks before Herod died. For the trip to be worthwhile, he probably went to Egypt before he was 2. But he did not flee to Egypt the night of his birth.
If nothing else, he remained in the Jerusalem area for eight days so he could be circumcised at the temple, according to Luke 2.
Following their time in Egypt, Joseph was afraid to return to Bethlehem, so he and Mary returned to her home town of Nazareth instead. It was a obscure, out of the way place where they felt they could raise Jesus without anyone seeing them. Nobody went to Nazareth. It was in the middle of nowhere.