Yeah, you use screen capture software, but it often requires a good bit of experimentation to get it right. Also, video encoding requires a good bit of experimentation to get a good mixture of quality, size and encode speed. Don't just accept defaults for anything. Tweak stuff until it works. If you don't know what it does, look it up.
I don't have any specific recommendations for screen capture in windows. There's fraps, I guess hypercam, camtasia and a few others. May as well try them all and see what works, however in a final video, watermarks are unacceptable. If you can't buy the software, crack it, or edit it out in post.
As for encoders, x264 is good. There are plenty of guides on configuring all the options and which are best. Probably best to stick with DXVA compliance if you want to make sure youtube will properly decode it. Another trick with youtube is that simple gameplay footage scaled up to 1080p will look a lot better on youtube than 480p or 360p, so make sure your videos are 1080p or at least 720p. This surprisingly won't impact the final size too much if you're just doing simple nearest or linear/bilinear/cubic/bicubic scale. Stay away from sinc/lanczos scaling for simple 2D graphics, however they should be OK for more photorealistic games.
EDIT: more info: You should really try to capture the video with the emulator running in a 2x nearest neighbor scale mode in a square/1:1 pixels mode, then later on in post, scale it down to 1/2, then nearest neighbor scale it up to the largest integer scale factor that fits within your target size, then linear scale the rest of the way. I would probably get and learn avisynth if I was really serious in to getting in to this.
Example:
SNES: original resolution: 256x224, Target aspect 4:3, target resolution 1440x1080
Emulator: 512x448 nearest neigbor
Capture: 512x448. Make sure your capture program can capture without chroma subsampling and doesn't do any lossy compression.
In post: original resolution: 512x448
Scale down: 256x224 nearest neigbor
Scale up: floor(1440 / 256) * 256 =1024, floor(1080 / 224) * 224 = 896, 1024x896, nearest neighbor
Scale up: 1440x1080, linear
____________________