I think that you should be able to put 2, 3, or 4 builders at one project to speed the building project up. This way you are making a sacrafice of loosing a builder for your project but now are able to finish the project fast. It also adds a strategic element to the building aspect of your villlage. I don't see any cons to the idea of being able to focus your builders on one project to speed the building time up. It would be fair in all ways because you are taking away a builder from another building project to speed up the other one.
Please let me know what you think about this idea to add to the game.