In short there is no way to stop any one from sharing your private videos be it on a custom solution or a public one like youtube.
Take Netflix for example, you pay £6 a month to watch as much content as you like, there is nothing to stop any one watching every thing on Netflix and recording it whilst they do so using a screen capture program and then sharing it with friends family w/e.
This is just one of those things, as far as getting the videos private so only those with a paid subscription on your site can view them (wont prevent them from recording it) then there are a tone of solutions out there for you, personally i would try to achieve what you are doing using some sort of system where a user registers on a webpage (they buy a subscription), Once they have the subscription the videos page would then be visible to them until such times as there subscriptions runs out. You could probably achieve this using a forum or wordpress :D
Hope this helps.