A 404 error on the web is what a web server responds with when it is tasked with serving up a resource that it can't find.
1. It should still look like your website
If you don't specifically tell your web server how to handle 404 errors, it will serve up a very plain and generic looking error page. That just says something like "Not Found - The requested URL was not found on this server." on a plain white page. This is an unhelpful and rather painful roadblock for a user. A 404 page should look like an error page, but it should still look like your website. If you are using a CMS, this is probably already handled for you. If you have a static site, you can specify a 404 template through your .htaccess file.
Nobody is trying to land on your 404. Most likely they came from a bad link either on your own site or out in the wild. They probably aren't very happy about it, so this is a golden opportunity to apologize and hopefully turn that frown upside down.
It's possible the page they are looking for still exists, it's just the URL has changed or it was incorrectly typed somewhere. Since your site should already have search anyway, this is a good opportunity to provide that search box front-and-center. This gives the user the opportunity to search for what they were trying to link to and dig up the correct URL for themselves.
4. Give readers useful links
Maybe the page they are linking to isn't broken, it just doesn't exist anymore. In addition to search, you may want to dish out some links they will find useful. Perhaps links to the homepage, the archives page, or other pages you already know are commonly visited / popular pages on your site?
5. Way to Contact / Report Error
It is possible a user landing on a 404 page is having some strong emotions about it. They could be very concerned, worried, or they could be down right ticked off. It is a good idea to give that user some way to release, and since you can't hand them a punching bag, a contact form might do the trick. This is, of course, not only good for them but good for you, as you can learn about the problem in order to fix it or otherwise prevent it from happening again.
6. Automatic Reporting
Having users help report errors for you through a contact form is great, but only a small percentage of users are likely to do that. An alternative is to have your 404 page automatically report the error to you. There are many ways you could do this. A quick-and-dirty way to do it would be to make your 404 page a PHP file, and use the mail function to send an email of the URL that was landed on.
<?php mail("email@example.com", "404 report", $_SERVER['REQUEST_URI'], "From: firstname.lastname@example.org\n") ?>
NOTE: The above code is probably fine for a small site but not very practical on high traffic sites. Writing the errors to a Database in those circumstances is probably more practical.
Even serious sites can get away with a little humor on the 404 page. This is definitely a good opportunity to loosen up a little and do something funny.
I'm on the fence about this one, but it's possible to automatically redirect a 404 error to another page. Perhaps the homepage displaying a message about the error? Perhaps a search page with keywords from the URL parsed out?
9. File Size
You might be surprised at the amount of malicious activity that your web server has to deal with. One thing that surprised me a while back was how many requests there were for "weird" files. Things like requesting the favicon in strange subfolders or looking for random images that don't exist. When these things don't turn up, the server loads up your 404 page. Needless to say, serving up a full webpage for all these requests adds up to some serious bandwidth. Your best bet is to fight against these malicious requests, but you should also take care to consider the total page size of your 404 page size and reduce it if bandwidth is a major concern.