Hex: Uses a mix of 6 numbers and characters
RGB: Uses 3 sets of 3 numbers which have a range of 0 – 255
Between the 2 the only main different is the syntax in which you use to generate the color, depending if your already using a pre-existing system that uses only one of the two it’s best to keep going with it otherwise it’s really up to you which one you use.
Tip: Personally i prefer hex but there are times with a need an alpha channel color so instead i would use RGBA (Red, Green, Blue and Alpha) which is the same as RGB but allows for a 0 – 1 alpha level as the forth argument. Browser compatibility is limited to IE9+ but all modern browsers support it with ease.