I think you are more than justified with the load time most people running a slow connection aren’t going to expect a gallery to load as fast as the google search page. I thought it loaded very quickly myself but I have a fast connection.
I noted the slider Idea in my last comment and just realized there is a good article on this site about it
If I were you I would try to implement this it would make your site stand out a bit more.
That said I really like how you design is as simple and lightweight as it is and still looks good. In my opinion you found a great balance here I just know there are so many designers that are very critical of load time.