Signing FF extensions is pretty straightforward. Lately I've been going through some strains related to them, so I thought I'd share lesson learned.
You have two sides that need to match with each other: online update URL and extension itself. Online update URL is a single file - update.rdf. Extension is an XPI file, with install.rdf file inside.
Update.rdf contains a list of available extensions. Each extension should have checksum information, e.g. in sha1 algorithm.
Once update.rdf is complete with all information and checksums, it must be signed with signer key.
Note that, every time you change any of extension, it's checksum will likely change and so you need to update and re-sign update.rdf as well.
On the other side, the install.rdf included in our XPI file needs to have our public key. Once you create install.rdf and put public key there, you can forget about it. XPI file does not have any checksum or signing.
This all sounds nice and well but sometimes it doesn't go as well. Let's suppose public key included in XPI file was wrong and the file is already in the public. Every update attempt will fail with error about malformed signature or updateKey. That's because during update, Firefox tries to match public key in XPI to signature in update.rdf.
The only solution is to manually install extension again, with correct public key. It's necessary to clear browser cache before performing manual install, because FF uses cache first if the extension is already there.
Detailed information about RDF file formats is here.