Different languages use different quotation marks. In English, is usually looks like this: “…”. In Germany it looks like this: „…“. And in French they use other symbols: « … ».
Correct typography is not always easy to get, especially when you can't type those symbols using your keyboard (even with the correct language set up). But where is a way to help you, at least in blockquotes.
CSS for quotes
You can define some CSS which would automatically put the correct quotes on a q element (usually inside of a blockquote element). There is a default to that property implemented in the browser. Just add this:
q { quotes: "\00ab" "\00bb" "\2039" "\203A"; }
This example defines the two variants of French quotes (as quotes can have "inner quotes" as well).
Using the HTML element
But there is an easier way. Just don't define anything specifically and let the browser handle the correct quoting. All you have to do is to set the correct lang attribute on the document.
<html lang="fr">
...
</html>
Now you can just use a q element and the quotation marks will match the defined language.
Fixing quotes in WordPress
WordPress is adding the lang attribute to the HTML tag, but unfotunately it is using the "four digit local", so instead of "fr" it is using "fr-FR" which will break the quoting for most browser. That's why I have written a small plugin to fix that.