Those Who Favor Protectionism are either Special Interests or the Tools of Them
A special interest is a group that mobilizes to pass laws for its benefits rather than society's. The worst special interests are those whose laws end up screwing society even though they benefit the special interest itself. Those who oppose trade fit clearly into this group.

Free trade is generally opposed because although it creates a net benefit for society, that benefit is spread out and difficult to notice, while the cost is concentrated allowing political mobilization.

Industry tariffs within the sugar, textile, and steel sectors (as three examples) or subsidies for farmers not to grow products create a net loss for our society through higher prices that must be borne by the consumer. This becomes even more true when coupled with the fact that these tariffs/subsidies often result in corresponding preferences in foreign countries that harm our workers that create exported products.

When you look at all of these factors, I don't see how anybody could AVOID casting those who oppose free trade as the tools of special interests.