The Sorites Paradox works in two directions.

Top-down: 1,000,000 grains of sand is a heap of sand; a heap of sand minus one grain is still a heap; therefore one or grain of sand (or zero!) is a heap of sand.

Bottom-up:
One grain of sand is not a heap of sand; a non-heap of sand plus one grain is still not a heap; therefore there are no heaps of sand.

Will Wilkinson’s defense of discrimination laws appeals to the absurdity of the Bottom-up version of the Sorites paradox:

I used to think that if negative rights to non-interference were
strictly observed, liberty was guaranteed, but I don’t now. Here’s how I
had thought about the matter.  One racist acting in a private capacity
on his or her racist beliefs can’t violate anyone’s legitimate, negative
rights. (No one is entitled to another’s good opinion!) Two racists
acting as private citizens on their racist beliefs can’t violate
anyone’s rights. Therefore, I inferred, thousands or millions of racists
acting non-coercively on their racist beliefs can’t coercively violate
anyone’s rights. I now think this is quite wrongheaded.

In my previous critique of Will’s argument, I said that “he was wrong to change him mind.”  After further reflection, though, I’ve concluded that he moved from one wrong position to another wrong position.

Will’s initial argument, as he now happily admits, was invalid.  The mere fact that one person’s negative opinion doesn’t impair your liberty does not show that many people’s negative opinions do not impair your liberty. 

But what if we switch from the Bottom-up Sorites argument against discrimination laws to a Top-down version?  Like so:

1. If everyone else on earth treated you badly without violating your person or property, it would not impair your liberty. 
2. If N people treating you badly without violating your person or property would not impair your liberty, then (N-1) people treating you badly without violating your person or property would not impair your liberty.
3. Conclusion: Regardless of how many people treat you badly without violating your person or property, it does not impair your liberty.

This is clearly a valid argument for the conclusion that Will once held, then abandoned.  But is it sound?  Most people, probably including Will, would call it absurd.  But is it?  Consider the following claims:

1. If everyone else on earth was gay, it would not impair your liberty.
2. If everyone on earth refused to be your friend, it would not impair your liberty.
3. If everyone on earth refused to buy a ticket to your poetry recital, it would not impair your liberty – even if poetry recital were your only marketable skill.
4. If everyone else on earth believed in astrology, it would not impair your liberty.
5. If everyone else on earth refused to work for you, it would not impair your liberty.
6. If everyone on earth refused to dine at your restaurant, it would not impair your liberty.
7. If every woman on earth refused to date you, it would not impair your liberty.

You don’t have to be a dogmatic libertarian to accept many or even all of these claims. 

What makes these claims broadly plausible?  The moralized theory of liberty that most of us share: If you don’t rightfully own something, you can’t legitimately complain that the way the rightful owners use it “impairs your liberty.”  Since you don’t own other people, you can’t legitimately complain that their sexual preferences impair your liberty.  Since your don’t own other people’s money, you can’t legitimately complain that how they spend their money impairs your liberty.  Etc.

The key difference between libertarians and normal people isn’t acceptance of this sort of claim.  The key difference, rather, is whether you accept such claims for employers.  E.g.:

1. If every employer on earth refuses to hire you, it does not impair your liberty.
2. If every employer on earth is mean to you, it does not impair your liberty.
3. If every employer on earth refuses to offer you healthcare, it does not impair your liberty.
Etc.

My question: Why on earth should we regard employers so differently?

Because you need their money to live?  You could say the same about customers in a world of self-employment.  If customers utterly disdain your wares, you’re in dire straits.  But have the world’s customers impaired your liberty?

Because you’ll be miserable if employers don’t give you a decent job?  You could say the same if everyone refused to be your friend.  You might even be suicidal.  But has the human race impaired your liberty?

It is easy to dismiss libertarians for “refusing to see” the coercive power of social pressure.  The reality, though, is that almost everyone usually “refuses to see” the coercive power of social pressure.  The simplest explanation is that people “refuse to see” social pressure as coercive as long as people have a moral right to exercise it.  You don’t have a moral right to a friend or a date or an audience, so you remain free despite your deprivation. 

The libertarian critique of discrimination law isn’t some weird novelty.  Once you stop treating employers as uniquely morally suspect, the libertarian critique immediately follows from common-sense morality.  The Sorites paradox, properly deployed, merely cements the libertarian case.