The sidewall number is a maximum, not a recommendation. It is the pressure the tyre is rated to hold safely. Running it is like driving at the top speed on a speedometer because it is printed there.
What actually sets your pressure is three things:
Total weight on the tyre. You, the bike, and anything you are carrying. Heavier means more pressure. This is the dominant factor and it is why one number cannot be right for everybody.
Tyre width. A wider tyre holds the same weight at much lower pressure because there is more air in it. This is most of why pressures have fallen so much - tyres have got wider.
Weight distribution. More weight sits on the rear than the front, so the rear wants more pressure. Usually a small but real difference.
And the reason lower is usually better, up to a point: a very hard tyre does not roll faster on real roads. On smooth laboratory surfaces it does; on actual tarmac, an over-inflated tyre bounces rather than deforming over the surface, and that costs both speed and comfort. There is a pressure below which losses rise again, and most people are nowhere near it - they are far above the point where it stopped helping.
So the harshness you feel is not the price of speed. It is just harshness.