If I were to emulate the effect I would use a combination of 3 jQuery plugins:
1. For the changing of images, when the thumbnails are clicked: http://razorjack.net/quicksand/
2. For the actual arrangement of the images: http://masonry.desandro.com/
3. For pop-out image:
any jQuery modal box plugin like colorbox, lightbox, etc.