Signing on Paper
This is my go-to recommendation for signing on light backgrounds. It comes in a variety of colors, makes nice fine lines, and is permanent and waterproof. (Generally anything labelled archival and waterproof will be good, but I like the evenness of the lines). On paper, even ball point pens are okay - I have seen ketubot from the seventies where all the text and design have faded to illegibility (shouldn't be a problem with my prints), but the signatures are still visible in ball point pen. Other pens will work as long as you stay away from felt tip markers (which can be blotchy).
Note that although it dries fast, it may take a little longer to dry than you expect. Be careful to let it dry!
(You may be able to find it for much cheaper in local art/office supply stores, but can't beat the convenience of getting it on Amazon - and I get a tiny commission).
Signing on Paper
On Paper, you have a little more latitude to be creative with pens. Choosing a dark background ketubah, means that you have to put a little extra thought into what you will sign with and what color. I highly recommend trying out any pen on paper that is a similar color.
Signing on Canvas & Metal
Choosing a non-traditional material means that you may have to be more flexible about the look of the signatures. The beauty and long-lasting colors make it worthwhile, but it is a good idea to have a realistic understanding of your materials. Canvas has a slick coating which protects it from damage but also may make signatures a little tricky. Metal prints also have a protective coating - not as slick as glass, but still a little tricky. Signatures are less likely to be as fine as on paper.
Use a fine point Sharpie. After experimenting and getting feedback from couples, these seem to work the best. The Sakura pens do work, but may bead and take a very long time to dry.
On Dark Backgrounds
It can be very tricky to find just the right thing to use on dark backgrounds. Generally you want something labelled 'permanent.' Paint pens are your best bet. Gel pens may work on canvas if there is a color you must have (try them out on paper, then the back of gallery canvas or on the very edge of the loose canvas). Be careful when choosing gel pens since old gel pens may make irregular lines.
Whatever you choose, have plenty of scratch paper on hand. I highly recommend a visit your local art supply store to try some pens out before you buy - most art supply stores will open samples for you. If there are none open, just explain what you need and ask for help!