It’s standard for WordPress, that when an image is used in a post and you add a caption, it will be wrapped in a DIV that has an inline style width that is 10 pixels more than the image it contains. I assume that’s your main issue.
So, if I post an image that is 100 pixels wide and I add a caption, WordPress will change the markup to this (simplified):
I would say your problem would be solved best if you would just style this code accordingly (get rid of the border, light grey background color and override that inline style width). Something like this:
width:auto !important; /* <-- A HA! */
As for the middle (un-captioned) image having an “awkward top border”, I assume you mean it has more top margin than all the others. This is because both the first image and the second have floats, and so the space between them is the sum of the bottom margin of the first and the top margin of the second: 20 pixels.
This space is NOT there between the second and the third image, because only the second image has a float (the third one doesn’t). Because of this, the space between them will not be the sum of the margins (bottom margin of 10 + top margin of 10), but the largest of the two; in this case, just 10 pixels.
This is quite theoretical stuff, but the point is that the (vertical) space between two items will be the SUM if they are both floated, or overlap (and result in the maximum of the two) if one (or none) are floated.