Whatever you choose, have plenty of scratch paper on hand. I highly recommend a visit your local art supply store to try some pens on similar color backgrounds before you buy - most fine art supply stores will open samples for you. If there are none open, just explain what you need and ask for help!
This is my go-to recommendation for signing on light background paper. It comes in a variety of colors, makes nice fine lines, and is permanent and waterproof. (Generally anything labelled archival and waterproof will be good, but I like the evenness of the lines). On paper, even ball point pens are okay - I have seen ketubot from the seventies where all the text and design have faded to illegibility (shouldn't be a problem with my prints), but the signatures are still visible in ball point pen. Other pens will work as long as you stay away from felt tip markers (which can be blotchy).
Note that although it dries fast, it may take a little longer to dry than you expect. Be careful to let it dry!
(You may be able to find it for much cheaper in local art/office supply stores, but can't beat the convenience of getting it on Amazon - and I get a tiny commission, something like $ 0.10 on each sale).
On Paper or Mounted Paper you have a little more latitude to be creative with pens. I highly recommend trying out any pen on paper that is a similar color before using it to sign on your ketubah.
Choosing a non-traditional material means that you may have to be more flexible about the look of the signatures. The beauty and long-lasting colors make it worthwhile, but it is a good idea to have a realistic understanding of your materials. Canvas has a slick coating which protects it from damage but also may make signatures a little tricky. Metal prints also have a protective coating - not as slick as glass, but still a little tricky. Signatures are less likely to be as fine as on paper.
For best results, use a fine point Sharpie.
After experimenting and getting feedback from couples, these seem to work the best.
It can be very tricky to find just the right thing to use on dark backgrounds. Generally you want something labelled 'permanent.' Paint pens are your best bet. Gel pens may work on canvas if there is a color you must have (try them out on paper, then the back of gallery canvas or on the very edge of the loose canvas). Be careful when choosing gel pens since old gel pens may make irregular lines.