question about internet bottlenecks

[pretentious]

When it comes to website loading times. It is pretty much down to the clients internet connection right? I figure the data center the site is hosted would use gigabit ethernet or fiber to the ISP(i know there is a technical term for this, but you get what i mean) and from there the hosting company would pay for an enterprise standard (BDSL or better?) agreement with the ISP to take it to the client and the client whos internet speed slows down loading. Am I right about this?
Wether my site loads fast is determined by wether the client has dial up or google fiber and nothing to do with me?

[limdis]

Watch this video, it's actually a pretty damn good ELI5 version of how it works along with a lot of background research closer to the end. 11 min but seriously it's worth the time.

[cyberdrain]

I thought it basically came down to server upload and client download speed, unless heavy computation or a difficult lookup (DNS, archive or similar) takes a lot of time.