Even if the flag was flown by an actual bona fide Nazi it would not have mattered to me.
1) Back in earlier days when people were more reasonable it was recognized that a small percentage of the population was extreme and we should not be hasty with generalizations and guilt by association.
2) The Left has perpetuated false flag versions of this sort of thing where they dress up as some hate group and pretend to be fans of an political adversary. This sort of thing was actually advocated in Saul Alinsky's "Rules for Radicals".
3) According to how free speech has been framed and how it should be protected it classically is about "standing up for the free speech of groups you disagree with and even loath". Basically that the KKK, neoNazi's etc. have free speech the rights to exist and congregate etc. just as long as they aren't doing one of the three things below:
a) committing actual physical violence, destroying property and similar kinds of felony crimes like kidnapping etc.
b) urging folks to do a)
c) secretly plotting to do a)
This has been the standard of how people looked at free speech and our other liberties before wokism, microaggressions etc. But the above is a necessary protection to prevent "Tyranny of the Majority".