My very favorite section from Mike Huemer’s The Problem of Political Authority begins by distancing himself from other libertarian philosophers:

The ideas of this chapter will strike many as too extreme and far too libertarian. Are we really forced to accept such conclusions? Surely, to arrive at these radical conclusions, I must have made some extreme and highly controversial assumptions along the way, assumptions that most readers should feel free to reject?

I am the first to say that libertarian authors have frequently relied upon controversial philosophical assumptions in deriving their political conclusions. Ayn Rand, for example, thought that capitalism could only be successfully defended by appeal to ethical egoism, the theory according to which the right action for anyone in any circumstance is always the most selfish action. Robert Nozick is widely read as basing his libertarianism on an absolutist conception of individual rights, according to which an individual’s property
rights and rights to be free from coercion can never be outweighed by any social consequences. Jan Narveson relies on a metaethical theory according to which the correct moral principles are determined by a hypothetical social contract. Because of the controversial nature of these ethical or metaethical theories, most readers find the libertarian arguments based on them easy to reject.

It is important to observe, then, that I have appealed to nothing so controversial in my own reasoning. In fact, I reject all three of the foundations for libertarianism mentioned in the preceding paragraph. I reject egoism, since I believe that individuals have substantial obligations to take into account the interests of others. I reject ethical absolutism, since I believe an individual’s rights may be overridden by sufficiently important needs of others. And I reject all forms of social contract theory, since I believe the social contract is a myth with no moral relevance for us…

Huemer then succinctly sums up the novelty of his approach:

The foundation of my libertarianism is much more modest: common sense morality. At first glance, it may seem paradoxical that such radical political conclusions could stem from anything designated as “common sense.” I do not, of course, lay claim to common sense political views. I claim that revisionary political views emerge out of common sense moral views. As I see it, libertarian political philosophy rests on three broad ideas:

i) First, a non-aggression principle in interpersonal ethics. Roughly, this is the idea that individuals should not attack, kill, steal from, or defraud one another, and in general, that individuals should not coerce one another, aside from a relatively narrow range of special circumstances.

ii) Second, a recognition of the coercive nature of government. When the state promulgates a law, the law is generally backed up by a threat of punishment, which is supported by credible threats of physical force directed against those who would disobey the state.

iii) Third, a skepticism of political authority as traditionally conceived. The upshot of this skepticism is, very roughly, that the state may not do what it would be wrong for any non-governmental person or organization to do.

Why should we accept these three broad ideas?

The main positive ethical assumption of libertarianism,
the non-aggression principle, is the most difficult to precisely
articulate. In truth, it is a complex collection of principles,
including prohibitions on theft, assault, murder, and so on. I cannot
completely articulate this principle or set of principles. Fortunately,
it is not the locus of disagreement between libertarians and partisans
of other political ideologies, for the “non-aggression principle,” as I
use the term, is simply the collection of prohibitions on mistreating
other individuals that are accepted in common sense morality. Almost no
one, regardless of political ideology, thinks that theft, assault,
murder, and so on are morally acceptable. It is not necessary to have a
complete list of these prohibitions, since the arguments for libertarian
conclusions have not depended upon laying claim to any such complete
list. It is also important to
understand that I am not making any
particularly strong assumptions about these ethical prohibitions. I am
not, for example, assuming that theft is never permissible. I am simply
assuming that it is not permissible under normal circumstances, as
dictated by common sense morality.

The second principle, that of
the coercive nature of government, is equally difficult to dispute. The
coercive nature of government is commonly forgotten or ignored in
political discourse, in which the justification for coercion is seldom
discussed. But virtually no one actually denies that the state regularly
relies upon coercion.

It is, then, the notion of authority that
forms the true locus of dispute between libertarianism and other
political philosophies: libertarians are skeptical about authority,
whereas most people accept the state’s authority in more or less the
terms in which the state claims it. This is what enables most to endorse
governmental behavior that would otherwise appear to violate individual
rights: non-libertarians assume that most of the moral constraints that
apply to other agents do not apply to the state.

Hence the final title of the book:

I have focused on defending skepticism about authority, by addressing
the most interesting and important theories of authority. In defending
this skepticism, I have, again, relied upon no particularly
controversial ethical assumptions. I have considered the factors that
are said to confer authority on the state, and found that in each case,
either those factors are not actually present (as in the case of
consent-based accounts of authority), or those factors simply do not
suffice to confer the sort of authority claimed by the state. The latter
point is established by the fact that a nongovernmental agent to whom
those factors applied would generally not be ascribed anything like
political authority. I have suggested that the best explanation for the
widespread inclination to ascribe authority to the state lies in a
collection of non-rational biases that would operate whether or not
there were any legitimate authorities. Most people simply never stop to
question the notion of political authority, but once we begin to examine
it carefully, the idea of a group of people with a special right to
command everyone else fairly dissolves.

These three ideas-the
non-aggression principle, the coercive nature of government, and
skepticism about authority-together demand a libertarian political
philosophy. Most government actions violate the non-aggression
principle-that is, they are actions of a sort that would be condemned by
common sense morality if they were performed by any non-governmental
agent. In particular, the government generally deploys coercion in
and for reasons that would by no means be considered adequate to
justify coercion on the part of a private individual or organization.
Therefore, unless we accord the state some special exemption from
ordinary moral constraints, we must condemn most government actions. The
actions that remain are just the ones that libertarians accept.

Disagree with the conclusion?  Huemer wants you to name a specific premise you reject:

might one avoid the libertarian conclusion? Only by rejecting one of
the three core principles I have identified. It seems to me extremely
unpromising to question the coercive nature of government, and I doubt
that any theorist will wish to take that tack. Some theorists will
question common sense morality. I have not undertaken a general defense
of common sense morality in this book, and I shall not do so now. Every
book must begin somewhere, and beginning with such assumptions as that
under normal conditions, one may not rob, kill, or attack other people,
seems to me reasonable enough. This is about the least controversial,
least dubious starting point for a book of political philosophy that I
have seen, and I think few readers will feel happy about rejecting it.

least implausible way of resisting libertarianism remains that of
resisting the libertarian’s skepticism about authority. I have addressed
what strike me as the most interesting, influential, or promising
accounts of political authority-the traditional social contract theory,
the hypothetical social contract theory, the appeal to democratic
processes, and appeals to fairness and good consequences. But I cannot
address every possible account of authority, and I suspect that a fair
number of thinkers will react to my performance by proposing alternative
accounts of authority.

This leads to his preemptive response to unaddressed criticism:

also suspect, however, that the general strategy I have relied upon
will be able to be extended to such alternative accounts. A theory of
authority will cite some feature of the state (taking “feature” very
broadly) as the source of its authority. My strategy begins by imagining
some private agent that possesses that feature… For instance, the
property of being something that would be agreed to by all reasonable
people, the property of being actually accepted by the majority of
society, and the property of producing very good consequences, are all
properties that a non-governmental organization, or the policies of such
an organization, could possess. As I say, then, we imagine a
non-governmental agent with the relevant feature. We then realize that
intuitively, we would not ascribe anything like political authority to
that agent. In particular, we would not ascribe it a comprehensive,
content-independent, supreme entitlement to coerce obedience from other
people. And so we conclude that the proposed feature fails as a ground
of political authority.

(Endnotes omitted.  I’m quoting from the draft, so there are slight differences from the final manuscript).

suspect that many readers of Huemer’s book will furrow their brows and
say, “That’s it?  That’s all you’ve got?”  But this perceived defect is
one of the book’s chief virtues.  Unlike almost every other political
philosopher, Huemer doesn’t waste your time.  He doesn’t try to
convince you of seven odd claims, then try to convince you that those
seven odd claims somehow imply his conclusion.  (See Rawls’ A Theory of Justice for an egregious example).  Huemer doesn’t try to make
readers feel intellectually inferior by making them learn a lot of
obscure jargon.  Instead, he clearly tells readers what he believes,
and why he believes it, and his conclusion follows directly from his
premises.  Readers of philosophy should settle for nothing less – or
more – than this.