According to Matthew, three magi brought Jesus gifts of gold, frankincense and myrrh. Gold had value then, like now, though it’s valuable today for somewhat different reasons. Though cheap now (~$2.50/oz.), at the time, frankincense was one of the most valuable commodities on the planet. Myrrh, likewise, was highly prized. So, what did Mary and Joseph do with these gifts? It seems like such valuable presents would have made them pretty well off, but they are never depicted as such.
Since Mary didn’t put out, I bet Joseph blew it all on hookers. Or, maybe they stashed it away, and gave it to Jesus as a bar mitzvah gift and he went gonzo with it, until he spent it all and had to wander around in robes. Maybe that’s why the lost years have been erased.