Well that was interesting. Let's sum up how miserable I was in the course of the last two days, and hope it can help some people not be as miserable (or laugh at me, who knows).

• Finding a way of accessing my OVH hosting though SSH: not as easy as it seems in their guide. They say one can indifferently connect to pro.ovh.net or to one's own domain? Well I didn't, ever, managed to log in pro.ovh.net. Had to use my domain name.
• Setting up wordpress: no problem, nice and easy. Uploading some files through Filezilla (since OVH won't let users access the internet from SSH), creating a database, launching a script. Not too hard.
• Setting up SVN: the OVH guide about that one is correct, at least. So you can follow it (french): http://guides.ovh.com/SVNMutu. A good thing to know: if (like me) you are some kind of barbarian and use Putty's exe file instead of installing it, "loading the key into Pageant" is not as simple as a double click, don't forget to map the "Putty key file" file extension to Pageant first. And, at least under Windows 7, the "icon in the lower right corner" does not appear, except in the notification area.
• Setting up websvn: no problem either. Again, uploading, unzipping, setting the svn root directory in the config.php file (which doesn't exist! You have to copy include/distconfig.php to create it).
• Adding authentication to websvn: now that was fun. Took me three hours. Every tutorial I stumbled upon used some Apache-specific features, which I apparently didn't have on my hosting. I finally quitted the idea of doing something clean, and made a very simple .htacess in the websvn root directory (four lines: AuthName, AuthType, AuthUserFile, Require valid-user) and a .htpasswd (which isn't in the accessible web directories of course ;)). The drawback is I can't filter acess by repository with this technique: but heck, it's better than having everyone viewing my SVN repositories.
• Finding out WebSVN has messed up with PHP version and broken WordPress and fixing it, i.e. adding SetEnv PHP_VER 5 at the beginning of WordPress' .htaccess.

Next step is integrating a bug tracker (I'm thinking of Mantis). We'll see how it goes, I have all the time in the world.