And here is where the argument for ideas as 'property' becomes absolutely absurd. If the first builder prevents the second from building, it can only be through depriving him of his physical property: his arms, his hatchet, his trees, his physical freedom, his life. So in order for intellectual property to be maintained, it must be more important than tangible, physical property -- it must be more real than the physical world itself.
Even after depriving the second builder of his ability to construct a shelter, the first builder cannot deprive him of the knowledge to build one, regardless of whether he discovered that knowledge on his own or witnessed it from the actions of the first. The idea exists in the head of the second builder, and no physical theft can deprive him of that thought short of brutal head-trauma. To claim that ideas can be owned by individuals in the same way that physical property can be owned is to claim that individuals can erase memories and knowledge from others.
Patents give ownership of ideas, as exercised in some form, to individuals and corporations. They exclude, by force, others from practicing the claims of the patent. This is the fundamental problem. Intellectual 'property' is in direct conflict with real, physical property. Labelling ideas as property necessarily diminishes property rights over real objects in the real world.
The same is true of the debate over copyright. If you own a pencil and paper, you own physical property. Copyright laws that restrict how you can use that pencil and paper necessarily conflict with the property rights naturally associated with that pencil and paper.
Calling ideas 'property' makes little sense. If you want to argue for giving ideas monopoly protection as an incentive for the act of thinking, fine. But let's call it like it is. Ideas are not private property, and the assertion that they can be maintained as such is ridiculous.
Patent law is a social contract. As long as patent law benefits society, it is reasonable to consider the use of patents as a practicality. But when patent law no longer benefits the society which has traded in their natural rights of creation and sacrificed their right to physical property in exchange, it is just as reasonable to consider fixing the law to bring about better balance.
Calling it "intellectual property" muddies the debate. A better term might be intellectual monopoly, or idea restriction, or C.R.A.P.