The abilities of ancient civilisations are quite stunning but then they didn't have the machinery we take for granted in this day and age to make our working lives easy. Even in the Victorian era, an Irish navvie working on a railway was expected to move twenty tons of 'muck' a day. Try it. You won't get close. Their diet was extraordinary, consisting of several meals a day with steak and lashings of beer.
The ancient Egyptians are a case in point. We often point to pyramids but they built other stuff too. Fort Buhen, now at the bottom of Lake Nasser since the sixties, had a circumference of more than a mile of thirty meter high walls and a dry moat. That's one substantial castle, even by medieval standards. Or perhaps Stonehenge, with larger monoliths dragged twenty miles to be uprighted, and smaller bluestones taken from quarries in South Wales before roads were constructed in Britain.
Bear in mind that when discussing Roman legions, they were expected to march with campaign gear and at the end of twenty or so miles a camp enclosure with a ditch and rampart was always dug (though loose stones were also used to build low walls in the middle east - re: Titus' campaign in Judaea in 72ad.
It is also worth pointing out that the Roman legions were being used as spare labour when not on campaign. Not because they were all expert engineers - they weren't, though they had capable men among them - but because a major project would otherwise require expensive recruitment of contractors and local labour. Always keep a military unit busy. Always.