This article is a step-by-step tutorial on how to convert an old overlay-based extension into a restartless (bootstrapped) extension that is also extractionless.
Requirements
First off, what kind of add-on are we talking about here? Well, XUL overlays and windows, JSM files, chrome & resource mappings with localization, default preferences, but no XPCOM components of your own. Some of that will have to be replaced and the rest will need to be loaded differently.
Next, what's the minimum version of Firefox we should require (preferably an ESR)? This guide targets Firefox 17 ESR or later (or anything else Gecko 17+, such as SeaMonkey 2.14+). This is two ESRs back (as of this writing), which should be plenty. Using the current Firefox ESR, stable version, or Nightly is generally a better idea if given the option, but some users take forever to upgrade.
There will be no usage of the Add-on SDK or any other external libraries here. Everything will use APIs available in Firefox 17+ or code provided here.
Step 1: Use Services.jsm
If you load one of Mozilla's internal JSM files, for example Services.jsm, you'll do so via privileged JavaScript code like this:
Components.utils.import("resource://gre/modules/Services.jsm");
From here on out, it is assumed you've imported Services.jsm somewhere at the top of whatever file you're in and will be using it in all code examples. The examples will also assume that you know how to properly add instructions to your add-on's chrome.manifest to add and remove resource, chrome, locale, & etc. mappings, so that you can access your files with custom paths such as:
resource://myAddon/filename.ext chrome://myAddon/content/filename.ext
Step 2: No more resource:// URIs for files internal to your bundle
Unfortunately, resource mappings in your chrome.manifest were not usable in restartless add-ons until Mozilla finally fixed this bug in Firefox 38, which looked bad, but only because Mozilla is still using resource:// URIs internally and in examples. Resource mappings for files in the mozilla distribution, such as Services.jsm (above), will continue to work. In overlay extensions, you can place a resource mapping in the chrome.manifest for your add-on and load your own JSM from resource:// URIs. It's a great way to modularize your code that's been available since Firefox 3. You can use chrome:// URIs with "Components.utils.import()" just fine; in fact you've been able to since Firefox 4. However, because it was implemented first for only file:// and resource:// but not chrome://, everyone who learned of this new feature learned that you had to load JSM from resource:// URIs and just stuck with that forever. It does still work if you don't have restartlessness to worry about, though the protocol (or scheme, or whatever term you prefer) really should be avoided at this point. The resource:// protocol actually bleeds into content which allows webpages to detect installed add-ons using the protocol, which is not particularly fantastic (just the static file contents, not any loaded script/data).
Step 2a: Load your JSM from chrome://
Now with that preface out of the way, this part is easy: drop support for Firefox 3.x if you haven't already, move your JSM files to wherever you've got your chrome mapping to for your XUL overlay and/or windows, import your files from that new chrome mapped path instead of the old resource one, and remove your "resource" line from your chrome.manifest file. It's probably a good idea to do this even if you aren't going fully restartless / extractionless due to the previously mentioned exposure to content of resource mappings.
Also, drop support for Firefox 4 through 9 while you're at it. Prior to Firefox 10, the chrome.manifest file you rely on wasn't loaded automatically for restartless add-ons. Hacks were required, and probably a bad idea.
Step 2b: Audit any remaining resource:// URI usage internal to your extension
If you don't need resource:// URIs for anything else, then you may be able to skip the next step. If not, see if you still can't do things any other way. As with JSMs, a chrome:// URI may be more appropriate. If you want to also make your add-on extractionless then you may need "step 3" if you're loading files with nsIFileInputStream or something similar, or a jar: URI might work. If not, a file:// URI might be fine for you. Restartless add-ons can easily get a URI for their install location on startup, so you should look into what you can do with that.
Step 3: No more nsIFile access for files internal to your bundle
For an extractionless extension, access to files internal to your bundle will not be possible using the nsIFile interface.
If you need to read data, or otherwise access files within your bundle, there are two options. The first is to use the nsIZipReader interface which permits continuing to use nsIInputStreams, etc. The second is to re-code to use XMLHttpRequest.
A file:// URI to the install location, or .xpi file, is available in installPath property of the bootstrap data structure passed to the startup(), shutdown(), install(), and uninstall() functions in what will be your bootstrap.js file (see below).
How to get and load the data of of your add-on's files using the Add-on Manager API:
// This is the OLD way of getting one of your files
const myAddonID = ...; // Just store a constant with your ID
Components.utils.import("resource://gre/modules/AddonManager.jsm");
AddonManager.getAddonByID(myAddonID,function(addon) {
var file = Services.io.newURI("resource://myAddon/filename.ext",null,null)
.QueryInterface(Components.interfaces.nsIFileURL)
.file;
var stream = Components.classes["@mozilla.org/network/file-input-stream;1"]
.createInstance(Components.interfaces.nsIFileInputStream)
.QueryInterface(Components.interfaces.nsISeekableStream);
stream.init(file, 0x01, 0444, 0); // read-only, read by owner/group/others, normal behavior
/* do stuff */
});
This bit of code is paraphrased and probably not to be recommended as-is, but it should work. (note that the usage of an octal integer literal, while standard for handling permissions, is dangerous and deprecated; usage of use ES5 strict mode to disable this and other foot-guns is recommended) If you need to read/manipulate binary data, a nsIBinaryInputStream instance is what you'll use on that stream (e.g. 32-bit integers, or fun stuff like 48-bit integers). Not ideal, but it works and performs more than sufficiently well. All of that code above is no longer viable if you also go extractionless (which you should).
Step 3a: Option 1: Use nsIZipReader
let zipReader = Components.classes["@mozilla.org/libjar/zip-reader;1"]
.createInstance(Components.interfaces.nsIZipReader);
zipReader.open(addonData.installPath);
...
From there you can open nsIInputStreams, extract files, or perform some other functions. Worst case would be that you extract a file to a temporary location and then use nsIFile operations upon the extracted file.
Step 3b: Option 2: Use XMLHttpRequest
Now, how do we replace that? The answer to that question is to load your file from a chrome:// URI using XMLHttpRequest. You may now have another question: wait, what does this have to do with XML or HTTP? The answer to that question is, of course, nothing. XMLHttpRequest is an API created by Microsoft, adopted by Mozilla and other vendors, and hacked into a Swiss Army knife of file loading. You can use it in a web page to fetch a file from your server and you can use it in your add-on to fetch a local file from your installation. The name is a vestigial structure that just makes things confusing. It is nonetheless the "Correct" and best way to do things. It's available in the global for a window, but in JSM you'll need to fetch it from an interface:
const XMLHttpRequest = Components.Constructor("@mozilla.org/xmlextras/xmlhttprequest;1",
"nsIXMLHttpRequest");
Here's how to load a file using it:
function loadFile(url,type,returnresult)
{
var request = new XMLHttpRequest();
request.open("GET", url, true); // async=true
request.responseType = type;
request.onerror = function(event) {
logErrorMessage("Error attempting to load: " + url);
returnresult(null);
};
request.onload = function(event) {
if (request.response)
returnresult(request.response);
else
request.onerror(event);
};
request.send();
}
loadFile("chrome://myAddon/content/filename.ext",dataType,function(data) {
/* do stuff with data */
});
Note: When using XMLHttpRequest to access a file:// URL the request.status is not properly set to 200 to indicate success. In such cases, request.readyState == 4, request.status == 0 and request.response will evaluate to true.
If your file is text, use "text" as your data type. If you're getting JSON this way make sure to explicitly set the type as "text" if you intend to parse it yourself. Even though it says that the default type is "text", Firefox will attempt to autodetect and fail, resulting in an error message in the console. This doesn't seem to break anything, but it is easily avoidable by being explicit with the type. MDN says you can set the type to "json" instead, if you prefer to have it parse things for you.
If your file is not text or JSON, then you're going to want to read binary data. The new way to do this is to use JavaScript typed arrays. Specify "arraybuffer" as your data type to get one from your XMLHttpRequest. To access that data you're going to need a data view to look at your typed array with. Data that's homogeneous might get away with using something like Uint32Array or one of the other standard typed array views, but it's probably a bad idea. The basic typed array views are not endian-safe. This is incredibly stupid. You'd think such an important new JavaScript feature made available for web content and chrome alike would at least have a way to set and keep track of endianness, but no, it doesn't. Don't use any of the basic typed arrays for any data you did not earlier write into them in the same program session. Also, they're not particularly helpful if your data isn't all of the exact same type (which it probably isn't).
The solution to read arbitrary binary data, of various sizes, in an endian-safe way, is to use DataView. The other typed array stuff is viable in Firefox 4+. This wasn't added until Firefox 15. If you were using nsIBinaryInputStream or anything similar, figuring out DataView will be fairly straightforward. Just read the docs and it's pretty simple. It will probably be notably faster than whatever you were doing before.
Reportedly XMLHttpRequest doesn't work reliably when used in JSM under versions of Firefox less than 16, however as previously mentioned, this guide should be taken as requiring Firefox 17+.
Step 4: Manually handle default preferences
Normal extensions load default preferences from a standardized file automatically. Restartless extensions don't (for no good reason). This part is fairly easy to implement yourself, at least. Here are some functions to handle this:
function getGenericPref(branch,prefName)
{
switch (branch.getPrefType(prefName))
{
default:
case 0: return undefined; // PREF_INVALID
case 32: return getUCharPref(prefName,branch); // PREF_STRING
case 64: return branch.getIntPref(prefName); // PREF_INT
case 128: return branch.getBoolPref(prefName); // PREF_BOOL
}
}
function setGenericPref(branch,prefName,prefValue)
{
switch (typeof prefValue)
{
case "string":
setUCharPref(prefName,prefValue,branch);
return;
case "number":
branch.setIntPref(prefName,prefValue);
return;
case "boolean":
branch.setBoolPref(prefName,prefValue);
return;
}
}
function setDefaultPref(prefName,prefValue)
{
var defaultBranch = Services.prefs.getDefaultBranch(null);
setGenericPref(defaultBranch,prefName,prefValue);
}
function getUCharPref(prefName,branch) // Unicode getCharPref
{
branch = branch ? branch : Services.prefs;
return branch.getComplexValue(prefName, Components.interfaces.nsISupportsString).data;
}
function setUCharPref(prefName,text,branch) // Unicode setCharPref
{
var string = Components.classes["@mozilla.org/supports-string;1"]
.createInstance(Components.interfaces.nsISupportsString);
string.data = text;
branch = branch ? branch : Services.prefs;
branch.setComplexValue(prefName, Components.interfaces.nsISupportsString, string);
}
Just grab the above, move your default preferences file to your chrome mapping, and then do the following line once during your startup:
Services.scriptloader.loadSubScript("chrome://myAddon/content/defaultprefs.js",
{pref:setDefaultPref} );
That's it. Once you've got the machinery to load and save preferences without having to jump through the various pref type hoops the actual preferences API sends you through, loading the actual preferences file is one line. I'd generally still recommend using the type specific functions for each pref individually, but to load the defaults just use the generic functions above and it's quite simple. The other generic functions are provided above in case you need them. Unfortunately, the built in APIs for dealing with preferences are missing this basic stuff, and its plain text handling doesn't work with Unicode properly.
Step 4a: Another way to handle default preferences
If you want to keep your preference file in defaults/preferences/, the approach above only works as long as your extension is unpacked. For packed extensions (the default), you can either load a module similar to Firebug’s prefLoader.js or load this workaround module (N.B. As of May 29, 2017, that module does not work with packed extensions; I'm preserving the link in case the author updates the gist to fix this issue).
Step 5: No more internal JAR files
You know how I've been mentioning extractionless add-ons every once in a while thus far? Well, you should probably consider switching to be extractionless when you go restartless. An old-style add-on installer is packaged something like this:
myAddon.xpi file (glorified ZIP)
└─ chrome.manifest
└─ install.rdf
└─ chrome folder
└─ myAddon folder
└─ content.jar file
└─ content folder (most files go here)
└─ locale folder (your locale files go here)
In versions of Firefox prior to 4.0 (Gecko 2.0), the XPI would be extracted into a folder in your profile's extensions folder. In current versions it stays unextracted as an XPI. If you were using input streams you already had to deal with this because they weren't an option without extraction. Opting-out to extractionlessness is done via the "unpack" flag in install.rdf.
Why the internal JAR? Well, two reasons:
- Prior to extractionless add-ons, all of your files got extracted. Putting them in one single JAR file made all your stuff load in one file read, which was faster. Extractionless XPIs are bascially a standardization of this idea.
- XPI files are glorified ZIPs, and ZIP compression is horrible. Doing anuncompressed internal JAR (aka, another ZIP) acts like a poor-man's solid archive and significantly boosts the overall compression ratio of the XPI, resulting in smaller installers and updates.
So, it's pretty much internal JARor extractionless XPI. Well, you can't use an internal JAR anymore. Firefox aggressively caches add-on data a bit too much. Your restartless add-on won't actually reload some types of files if they are in a JAR and the add-on is updated without a restart. The big culprits are JSM files and locale files (namely property files), though in some situations this is true for dynamically loaded image files too. You're still going to have to manually clear the chrome cache on add-on shutdown to work around this, but that doesn't seem to be enough with an internal JAR. So, time to switch to extractionless, too. See here for the list of stuff you can't have in addition to no resource:// URIs or file:// URIs to files inside your XPI.
If you actually can't find a way to go fully extractionless, you could hack together some combination of internal JAR(s) and extracted files. It can be done. However, you really should go extractionless. Firefox profiles aren't the pristine environment they're supposed to be. Software that pretends to be designed to protect security or privacy that some users have installed will sometimes delete files. There have been plenty of reports of add-on franken-installs with files of two versions mixed together. This might be due to malware or a bug in Firefox. In any case, I have noticed a significant improvement in reliability by going fully extractionless. Installing and updating a single file is far more idiot-proof.
Step 6: No more XUL overlays
Ok, now we're getting into some more drastic changes. You won't be able to use your chrome.manifest to load XUL overlays anymore with a restartless add-on. You could look into dynamically loading and unloading your overlay, however dynamically manipulating the DOM of your XUL window is usually the more straightforward route.
Figure out what XUL elements you need to create for your add-on to add your interface, where it needs to go into a XUL window, and how to do it. Docs: document.getElementByID(), document.createElement(), Element reference, Node reference (DOM elements are also nodes).
You'll need to write two functions. One to take a XUL window object and then create and add your elements, and then another to find your elements and remove them from the window object. The former will need to be run on add-on startup and the later on add-on shutdown. Until you get your bootstrap.js running you should use a basic overlay onto the XUL window with an event listener for "load" to catch overlay load and then run your manual UI construction function.
Step 6a. Details on adding elements dynamically to chrome XUL window
There is a way that makes constructing of UI a lot similar to the way it was made with XUL overlay. It involves using firebug.sdk. The next is example of the code:
var overlay =
TOOLBARBUTTON(toolbarButtonAttrs,
PANEL({'id': 'thepanel', 'type': 'arrow'},
HBOX({'align': 'start'},
VBOX(
HBOX({'class': 'pixel-hbox'},
DESCRIPTION({'value': this.stringBundle.GetStringFromName('firexPixel.opacity')}),
HTMLINPUT({'id': 'opacity-range', 'type': 'range', 'min': '0', 'max': '10'})
),
HBOX({'id': 'pixel-coords', 'class': 'pixel-hbox'},
LABEL({'control': 'coord-x', 'value': 'X:'}),
TEXTBOX({'id': 'coord-x', 'class': 'coord-box', 'placeholder' : '0'}),
LABEL({'control': 'coord-y', 'value': 'Y:'}),
TEXTBOX({'id': 'coord-y', 'class': 'coord-box', 'placeholder': '0'})
...
That way you build elements hierarchy with not much interaction with DOM, plus you can see tag properties and it children in a nice, structured way, just like in overlay.xul. You can find working example here. It involves using of firebug.sdk xul.js with few modifications.
Step 7: Manually handle global CSS Stylesheets
Any Global CSS style sheets which you are using will need to be registered upon load and unregistered when your extension is unloaded. Any CSS files used in any of your own XUL files will function normally without any extra work needed.
Components.utils.import("resource://gre/modules/Services.jsm");
var styleSheets = ["chrome://myExtension/skin/myStyleSheet.css"];
function startup(data,reason)
{
...
// Load stylesheets
let styleSheetService= Components.classes["@mozilla.org/content/style-sheet-service;1"]
.getService(Components.interfaces.nsIStyleSheetService);
for (let i=0,len=styleSheets.length;i<len;i++) {
let styleSheetURI = Services.io.newURI(styleSheets[i], null, null);
styleSheetService.loadAndRegisterSheet(styleSheetURI, styleSheetService.AUTHOR_SHEET);
}
...
}
function shutdown(data,reason)
{
...
// Unload stylesheets
let styleSheetService = Components.classes["@mozilla.org/content/style-sheet-service;1"]
.getService(Components.interfaces.nsIStyleSheetService);
for (let i=0,len=styleSheets.length;i<len;i++) {
let styleSheetURI = Services.io.newURI(styleSheets[i], null, null);
if (styleSheetService.sheetRegistered(styleSheetURI, styleSheetService.AUTHOR_SHEET)) {
styleSheetService.unregisterSheet(styleSheetURI, styleSheetService.AUTHOR_SHEET);
}
}
...
Step 8: Window icons
Firefox does not scan the chrome/icons/default directory of restartless or extrationless extensions for window icons. If you are using custom window icons, they will need to be moved to %MozDir%/icons/default/ upon load of your extension. Removal upon unload is not required, but you must be able to handle overwriting them upon load. This is because your unload() will not always be called prior to an upgrade (e.g. upgrade might take place while the application is not running). Further, this is a generic location for icons and the icon may still be in use by a different profile. Thus, you will probably want to use version numbers in the icon name (the ID of the window for which the icon exists).
Step 9: bootstrap.js
A bootstrap.js file in the root of your XPI, next to your chrome.manifest and install.rdf, will be the heart of your restartless add-on. Think of it as main.c, but for JavaScript based Firefox restartless add-ons. A basic bootstrap.js file:
Components.utils.import("resource://gre/modules/Services.jsm");
function startup(data,reason) {
Components.utils.import("chrome://myAddon/content/myModule.jsm");
myModule.startup(); // Do whatever initial startup stuff you need to do
forEachOpenWindow(loadIntoWindow);
Services.wm.addListener(WindowListener);
}
function shutdown(data,reason) {
if (reason == APP_SHUTDOWN)
return;
forEachOpenWindow(unloadFromWindow);
Services.wm.removeListener(WindowListener);
myModule.shutdown(); // Do whatever shutdown stuff you need to do on addon disable
Components.utils.unload("chrome://myAddon/content/myModule.jsm"); // Same URL as above
// HACK WARNING: The Addon Manager does not properly clear all addon related caches on update;
// in order to fully update images and locales, their caches need clearing here
Services.obs.notifyObservers(null, "chrome-flush-caches", null);
}
function install(data,reason) { }
function uninstall(data,reason) { }
function loadIntoWindow(window) {
/* call/move your UI construction function here */
}
function unloadFromWindow(window) {
/* call/move your UI tear down function here */
}
function forEachOpenWindow(todo) // Apply a function to all open browser windows
{
var windows = Services.wm.getEnumerator("navigator:browser");
while (windows.hasMoreElements())
todo(windows.getNext().QueryInterface(Components.interfaces.nsIDOMWindow));
}
var WindowListener =
{
onOpenWindow: function(xulWindow)
{
var window = xulWindow.QueryInterface(Components.interfaces.nsIInterfaceRequestor)
.getInterface(Components.interfaces.nsIDOMWindow);
function onWindowLoad()
{
window.removeEventListener("load",onWindowLoad);
if (window.document.documentElement.getAttribute("windowtype") == "navigator:browser")
loadIntoWindow(window);
}
window.addEventListener("load",onWindowLoad);
},
onCloseWindow: function(xulWindow) { },
onWindowTitleChange: function(xulWindow, newTitle) { }
};
As mentioned above, Components.utils.unload() will not work properly if the JSM file it is unloading is in a JAR. Also make sure to only unload your own JSM files to avoid accidentally breaking things horribly.
For tearing down and cleaning up on a per-window basis, there is another route you can take. Instead of directly calling your tear down function, make your unloadFromWindow() something like this:
function unloadFromWindow(window)
{
var event = window.document.createEvent("Event");
event.initEvent("myAddonName-unload",false,false);
window.dispatchEvent(event);
}
In each window you can then register on startup to listen for your custom "myAddonName-unload" event and just tear down and clean up when that event or a regular "unload" event comes in.
Step 10: Bypass cache when loading properties files
The above will get you a working add-on that will install without a Firefox restart. It will even get you a working add-on that will update without a Firefox restart... usually. Some parts work only if you don't look too closely; localization is one of them. As mentioned in the previous section, you'll need to clear the chrome caches on add-on shutdown, namely for chrome images and properties files. Doing this will get an update's new properties file to load, however sometimes this will instead produce an error on the next property access. It just doesn't seem that it can reliably clear the cache correctly, for whatever reason. String changes seem to be fine, however the addition or removal of strings can sometimes produce this error. It's not reliably reproducible, but it does happen. Yes, this is a pain in the ass.
The suggestion that seems to work is to use a hack to bypass the string bundle cache. You should still be caching a reference to your string bundle on add-on startup, preferably using XPCOMUtils.jsm to lazily load the file. For example:
Components.utils.import("resource://gre/modules/Services.jsm");
Components.utils.import("resource://gre/modules/XPCOMUtils.jsm");
XPCOMUtils.defineLazyGetter(this, "strings", function() {
return loadPropertiesFile("chrome://myAddon/locale/mystrings.properties");
});
function loadPropertiesFile(path)
{
/* HACK: The string bundle cache is cleared on addon shutdown, however it doesn't appear to do so reliably.
Errors can erratically happen on next load of the same file in certain instances. (at minimum, when strings are added/removed)
The apparently accepted solution to reliably load new versions is to always create bundles with a unique URL so as to bypass the cache.
This is accomplished by passing a random number in a parameter after a '?'. (this random ID is otherwise ignored)
The loaded string bundle is still cached on startup and should still be cleared out of the cache on addon shutdown.
This just bypasses the built-in cache for repeated loads of the same path so that a newly installed update loads cleanly. */
return Services.strings.createBundle(path + "?" + Math.random());
}
Just do strings.GetStringFromName(stringID) as you normally would. The lazy getter magic will cause the file to be automatically loaded the first time it is needed, after which point a reference to the loaded string bundle will be stored in "strings" for future accesses. You still need to clear the cache on add-on shutdown, however it will now also load cleanly on add-on updates. The old file should still be cleared.
Put it all together
That should be all the pieces. Your chrome.manifest will have just chrome and locale (and possibly skin) mappings in it now. No resource mappings or chrome overlays. The new entry point for your add-on is via bootstrap.js:startup() rather than a "load" handler in a XUL overlay.
Your localization handling should be unaffected by your transition to a restartless/extractionless add-on so long as you properly clear the chrome cache on add-on shutdown and load your properties files using the method listed above. Your property files and DTD files loaded from chrome:// URIs should work just as before. This is all assuming a minimum version of Firefox 17+ (or other Gecko 17+ application) which you should remember to state explicitly in your install.rdf.
Just remember that whatever you start you also need to have the ability to undo. In order for your add-on to reliably update without a restart it needs to be able to shutdown/disable cleanly.
Also note that once you do get this all up and running, your users will still have to restart Firefox once to install your first restartless update. While your new add-on may not need a restart to install, if you're updating from an old version that is not restartless then it will need a restart touninstall that first.
This tutorial was originally written by Dave Garrett from his experience porting the Flagfox extension.
Further reading
- Author original article.
- Another real-world example of porting overlay-based extension into restartless (git diff).
- Another example of using firebug.sdk xul.js to construct ui.