Extending a Protocol

Quick Start: Extending a Protocol

This tutorial implements a simple ping-pong style IPDL protocol, which sends a message from the content process (main thread) to the chrome process (UI Thread). The tutorial is designed for beginners and will walk you through all the steps you need to get things working.

The tutorial is designed for browser engineers who are implementing DOM/Web APIs that need to, for example, send a message to the OS or spin up something off the main thread - so it's biased towards supporting W3C/WHATWG DOM APIs.

What are we going to implement?

tl:dr: you can see the final implementation of PEcho.ipdl as a diff in Phabricator.

To make this tutorial less abstract (more fun!), we are going to implement a JS method called echo() on the Navigator interface. The method will take one argument, a DOMString, which we will pass to the parent process. Then we will send that same string back to the child process.

The WebIDL for this will be:

partial interface Navigator {
  Promise<DOMString> echo(DOMString someString);
};

Learning goals are:

Extending an existing IPDL protocol.
Simple message passing.
Working with MozPromises and JS promises.
Managing child actors with reference counting.

Visually, it's going to look something like this (except for the Operating System bit, which we are not actually going to do - just to illustrate what we could do):

IPDL tree, who manages who

Let's start by implementing the WebIDL above.

Implementing the `navigator.echo()`

In your favorite editor, open dom/webidl/Navigator.webidl

At the end of the file, add:

partial interface Navigator {
  [Throws]
  Promise<DOMString> echo(DOMString aString);
};

Now we need to implement the Echo() method in C++, so open up ./dom/base/Navigator.h and let's add the method definition, so under public::
```
already_AddRefed<Promise> Echo(const nsAString& aString, ErrorResult& aRv);
```
We use nsAString& as the DOMString comes in from JS as UTF-16. This will matter later, as we will need to convert it to UTF-8 (using nsCString) to send it to the parent process.

Now let's implement the method in ./dom/base/Navigator.cpp - we add a little bit of error boilerplate stuff, as is customary:

already_AddRefed<Promise> Navigator::Echo(const nsAString& aString, ErrorResult& aRv) {
  if (!mWindow || !mWindow->GetDocShell()) {
    aRv.Throw(NS_ERROR_UNEXPECTED);
    return nullptr;
  }

  RefPtr<Promise> echoPromise = Promise::Create(mWindow->AsGlobal(), aRv);
  if (NS_WARN_IF(aRv.Failed())) {
    return nullptr;
  }
  // Message passing magic will happen here!
  return echoPromise.forget();
}

Ok, let's ./mach build and make sure everything is compiling ok. Once it's compiled, ./mach run and you should be able to spin up the developer console and type:
```
navigator.echo("hi!");
```
That will return a pending Promise that never settles. Our goal is now to get that to settle with the value we sent in.

IPDL basics

While you are compiling, let's cover some IPDL basics.

In IPDL there two kinds of "actors" - a "parent" and "child" - these actors are literally implemented as C++ classes, so they are more than just abstract ideas.

Usually, a child wants to send messages to the parent to do something off main thread for us. And we generally want the parent to let a child know when some work is done - and send some confirmation or result data back to the child.

Sometimes, the parent will want to notify all its children of something. So communication can go in both directions. In this tutorial, we are just going to cover a child talking to a parent, and the parent responding with some data.

In order to manage parent-child relationships, IPDL uses a tree herirachy to manage the creation of children. As DOM APIs have access to the Window object, we generally want to use the "PWindowGlobal" protocol to manage things for us: this protocol is convinient because we can easily access it from the Window object... so as long as we can get a hold of a Window object, we can get it to do useful things for us (as we shall soon see!).

In our case, our protocol, which we will call PEcho.ipdl, will send a message to a parent, and the parent will respond with the message we sent. As a heirarchy, this will look like this:

IPDL tree, who manages who

This means that, PWindowGlobal.ipdl:

Manages: meaning that PWindowGlobal.ipdl is responsible managing the creation of instances of PEcho children and the parent. And the inverse is also true, as you will soon see: PEcho.idpl will need explicitly state that PWindowGlobal.ipdl is managing it.
Includes: this is kinda like "#includes" in C++, except it's not a preprocessor directive. It just includes the defintion and makes it available to PWindowGlobal.ipdl.
Constructs: by which we mean, PWindowGlobal is in charge of contructing the parent/child relationship. Without this, we can't send messages.

Anyway, let's put the above into practice by creating our custom protocol: PEcho.ipdl.

Custom protocol - PEcho.ipdl

This next part requires us to both specify the protocol and actually implement everything in C++. It's quite involved and, unfortunately, all steps need to be done before we can successfully recompile. We will try to compile our code as we go, noting what we need to do along the way.

To be sure, IPDL looks like a crazy mix of WebIDL, C++ header files, and ES6. The syntax should hopefully be fairly obvious though. Unfortunately, it's not really fully documented anywhere - and the IPDL parser gets easily confused (hence us excluding comments below).

Creating the protocol PEcho.ipdl

Let's start by saving the following file to ./dom/ipc/PEcho.ipdl. We will explain what all the parts mean below. Note that the file's name and the protocol name must match.

include protocol PWindowGlobal;

namespace mozilla {
namespace dom {

async refcounted protocol PEcho
{
manager PWindowGlobal;

parent:
  async Echo(nsCString data) returns (nsCString aResult);
  async __delete__();
};

} // namespace dom
} // namespace mozilla

Now, edit "./dom/ipc/moz.build" file and add 'PEcho.ipdl', to the IPDL_SOURCES array. Remember that it needs to be in alphabetical order or it won't compile.

Breaking it down:

include protocol PWindowGlobal; - We include PWindowGlobal, because it's going to manage the protocol for us.
async refcounted protocol PEcho {....} - This protocol is "async", meaning we are going to use MozPromises. It's also "refcounted", meaning it we can use RefPtr<> for instances that can be cycle collected.
manager PWindowGlobal; - We explicitly state that PWindowGlobal is our manager.
parent: - These are the messages that the parent is able to recieve. Note that the child doesn't need to receive any messages. This is because of "async" and "returns" forms a MozPromise.
async Echo(nsCString data) returns (nsCString aResult); - The "returns" here translates to a MozPromise, with a resolver that will settle a promise with the expected value.
async __delete__(); - This is called when an instance is deleted. It lets you do cleanup if needed.

Ok, if we now try to compile this (./mach build), it will say:

|manager| declaration in protocol `PEcho' does not match any |manages| declaration in protocol `PWindowGlobal'

That's good! It means the syntax is correct, and now we need to PWindowGlobal.ipdl to manage our PEcho protocol.

Updating PWindowGlobal.ipdl

Open up ./dom/ipc/PWindowGlobal.ipdl.

We need to add the following things:

At the top of the file: include protocol PEcho;
After "manager PBrowser or PInProcess;", add: manages PEcho;
Finally, under parent: add async PEcho(); - this is the "constructor" part we talked about before, which will allow us to eventually send messages.

So roughly, it looks like this (things in bold is what we added).

// Stuff..

include protocol PEcho;

async protocol PWindowGlobal
{
  // stuff...

  manages PEcho;

  child:
   // stuff...

  parent:
   async PEcho();

  // more stuff...
}

If the above additions are unclear, please see the final implementation of PEcho as a diff in Phabricator - search for "PWindowGlobal.ipdl".

Congrats! We've now set up the bi-directional relationship between PWindowGlobal and PEcho at the protocol level. We've told Gecko who manages who, and what messages we want to send.

Now we actually need to implement the "EchoChild" and "EchoParent" actors to handle sending and receiving messages. And we also need to implement the actual management of these objects by dom/ipc/WindowGlobalChild.{h|cpp} and dom/ipc/WindowGlobalParent.{h|cpp}.

If we now do ./mach build, it complains that "fatal error: 'mozilla/dom/EchoChild.h' file not found". That gives us a good place to start.

Defining EchoChild.h

Create ./dom/ipc/EchoChild.h, and define it as follows - the inline comments describe what's going on:

#ifndef mozilla_dom_EchoChild_h
#define mozilla_dom_EchoChild_h

// We include the protocol, which is automatically generated for us.
#include "mozilla/dom/PEchoChild.h"

namespace mozilla {
namespace dom {

// Note that EchoChild extends the protocol implementation.
// the protocol implementation will give us the actual
// methods for sending messages, as we will soon see.
class EchoChild final : public PEchoChild {
  friend class PEchoChild;

  // Allows us to participate in reference counting
  NS_INLINE_DECL_THREADSAFE_REFCOUNTING(EchoChild, final)

 public:
  explicit EchoChild();

 protected:
  void ActorDestroy(ActorDestroyReason aWhy) override;

  // We get SendEcho(nsCString& aString) from the superclass.

 private:
  ~EchoChild() = default;
  bool mActorAlive;
};

}  // end of namespace dom
}  // end of namespace Mozilla

And add the EchoChild.h to dom/ipc/moz.build, as part of the EXPORTS.mozilla.dom array. Again, remember that this must be in alphabetical order!

Compiling this will now complain: "fatal error: 'mozilla/dom/EchoParent.h' file not found", so let's define EchoParent.h.

Defining EchoParent.h

Like before, create ./dom/ipc/EchoParent.h, and code it as follows - the inline comments describe what's going on. Note the RecvEcho(), we finally get to define where a message is received, yay!:

#ifndef mozilla_dom_EchoParent_h
#define mozilla_dom_EchoParent_h

#include "mozilla/dom/PEchoParent.h"

namespace mozilla {
namespace dom {

class EchoParent final : public PEchoParent {
  friend class PEchoParent;

  NS_INLINE_DECL_THREADSAFE_REFCOUNTING(EchoParent, final)

 public:
  EchoParent();

 protected:
  // We receive messages via Recv* prefixed methods!
  mozilla::ipc::IPCResult RecvEcho(const nsCString& aString,
                                   EchoParent::EchoResolver&& aResolver);

  mozilla::ipc::IPCResult Recv__delete__() override;

  void ActorDestroy(ActorDestroyReason aWhy) override;

 private:
  ~EchoParent() = default;
  bool mActorAlive;
};

}  // end of namespace dom
}  // end of namespace mozilla

#endif

Now, and add the EchoParent.h to dom/ipc/moz.build, as part of the EXPORTS.mozilla.dom array. Again, remember that this must be in alphabetical order!

Ok, if we now try to compile, it should complain that "error: no member named 'AllocPEchoParent' in 'mozilla::dom::WindowGlobalParent'". This is part of WindowGlobalParent managing our protocol for us. So let's add that.

Defining WindowGlobalParent's Alloc/DeallocPEchoParent()

Open up dom/ipc/WindowGlobalParent.h and we are going to add two things:

At the top, add #include "mozilla/dom/EchoParent.h"
And then, as part of the WindowGlobalParent class definition, add the following two public methods:
```
already_AddRefed<EchoParent> AllocPEchoParent();

bool DeallocPEchoParent(PEchoParent* aActor);
```

Natrually, we will have to do the Alloc/Dealloc dance for the children too. So, let's do that now.

Defining WindowGlobalChild's Alloc/DeallocPEchoChild()

Same as above, now open up dom/ipc/WindowGlobalChild.h and we are going to add two things:

At the top, add #include "mozilla/dom/EchoChild.h"
And then, as part of the WindowGlobalChild class definition, add the following two public methods:
```
already_AddRefed<EchoChild> AllocPEchoChild();

bool DeallocPEchoChild(PEchoChild* aActor);
```

Super tedious, we know... nearly there though!

Let's give this a compile and see how we are doing! We should see ./mach build make significant progress now, but obviously, wihtout a backing cpp implementation, it's not going to do much.

Let's implement the .cpp files, starting with WindowGlobalParent.cpp.

Implementing methods on WindowGlobalParent.cpp

Open up dom/ipc/WindowGlobalParent.cpp and let's implement:

AllocPEchoParent()
DeallocPEchoParent(PEchoParent* aActor)

So, let's add:

already_AddRefed<EchoParent> WindowGlobalParent::AllocPEchoParent() {
  puts("WindowGlobalParent::AllocPEchoParent was called");
  RefPtr<EchoParent> actor = new EchoParent();
  return actor.forget();
}

bool WindowGlobalParent::DeallocPEchoParent(
    PEchoParent* aActor) {
  RefPtr actor =
      dont_AddRef(static_cast(aActor));
  return true;
}

The "puts()" there will help us see what's going on once we get things going. Naturally, we want to remove that once we done implementing.

Implementing methods on WindowGlobalChild.cpp

So, now we need to do the child.

Open up dom/ipc/WindowGlobalChild.cpp and let's implement:

AllocPEchoChild()
DeallocPEchoChild(PEchoChild* aActor)

already_AddRefed<EchoChild> WindowGlobalChild::AllocPEchoChild() {
  RefPtr<EchoChild> actor = new EchoChild();
  return actor.forget();
}

bool WindowGlobalChild::DeallocPEchoChild(PEchoChild* actor) {
  delete actor;
  return true;
}

And now finally, let's implement EchoChild.cpp and EchoParent.cpp.

Implementing EchoChild.cpp

This is a little one. Create dom/ipc/EchoChild.cpp and implement:

#include "EchoChild.h"

namespace mozilla {
namespace dom {

EchoChild::EchoChild() : mActorAlive(true) {}

void EchoChild::ActorDestroy(ActorDestroyReason aWhy) {
  mActorAlive = false;
}

}  // end of namespace dom
}  // end of namespace mozilla

Ok, now add 'EchoChild.cpp' to the UNIFIED_SOURCES of dom/ipc/moz.build. Remember alphabetical order for the 1000th time!

Implementing EchoParent.cpp

And now, we can finally implement dom/ipc/EchoParent.cpp. This is the exciting one, as it the one that actually receives the IPC message via RecvEcho().

#include "EchoParent.h"

namespace mozilla {
namespace dom {

EchoParent::EchoParent() : mActorAlive(true) {}

mozilla::ipc::IPCResult EchoParent::RecvEcho(
    const nsCString& aString, EchoParent::EchoResolver&& aResolver) {
  puts("[EchoParent] RecvEcho() - yay!");
  aResolver(aString); // Let's send it back!
  return IPC_OK();
}

mozilla::ipc::IPCResult EchoParent::Recv__delete__() {
  mActorAlive = false;
  return IPC_OK();
}

void EchoParent::ActorDestroy(ActorDestroyReason aWhy) {
  mActorAlive = false;
}

}  // end of namespace dom
}  // end of namespace mozilla

Ok, now add 'EchoParent.cpp' to the UNIFIED_SOURCES of dom/ipc/moz.build. Remember alphabetical order, again... sorry!

This should compile now, so run ./mach build... but it doesn't yet do anything. We need to now got back to Navigator.cpp and get it to send the message for us!

Back to Navigator

Ok, we now have all the things we need to send a message and get a response, we just need to put it all together.

This part can be a bit confusing, so if in doubt see the final implementation of PEcho as a diff in Phabricator. Just look for Navigator.h and Navigator.cpp.

Additions to Navigator.h

Firstly, let's make sure we create the child actor (EchoChild) instance, which will allow us to send the message. In creating the instance, we will also make sure the right "Alloc" magic happens, so we can actually send messages.

To dom/base/Navigator.h, let's add the following method and member, along with the other members. So, essentially, the stuff to add in bold:

// find this:
namespace mozilla {
namespace dom {
// add:
class EchoChild;

// Make this private:
EchoChild* GetEchoChild(ErrorResult& aRv);

// other stuff already in the file... scroll to the end and add private member:
RefPtr<EchoChild> mEchoChild;
} // namespace dom
} // namespace mozilla

Navigator.cpp - GetEchoChild()

Now, add the following includes at the top of the file:

#include "mozilla/dom/EchoChild.h"
#include "mozilla/dom/WindowGlobalChild.h"

To the "NS_IMPL_CYCLE_COLLECTION_UNLINK_BEGIN(Navigator)" we can now tell Gecko that mEchoChild can participate, so let's add:

NS_IMPL_CYCLE_COLLECTION_UNLINK(mEchoChild)

Now, the GetEchoChild() private method will do the intial setup for us.

EchoChild* Navigator::GetEchoChild(ErrorResult& aRv) {
  if (!mEchoChild) {
    if (!mWindow) {
      aRv.Throw(NS_ERROR_UNEXPECTED);
      return nullptr;
    }
    auto windowGlobalChild = mWindow->GetWindowGlobalChild();
    // Let's get us a child!
    mEchoChild = windowGlobalChild->AllocPEchoChild();
    // Let's make sure it's initialized and ready to send messages.
    windowGlobalChild->SendPEchoConstructor(mEchoChild);
  }
  return mEchoChild;
}

And finally... we can now finally finish off ::Echo():

already_AddRefed<Promise> Navigator::Echo(const nsAString& aString,
                                          ErrorResult& aRv) {
  if (!mWindow || !mWindow->GetDocShell()) {
    aRv.Throw(NS_ERROR_UNEXPECTED);
    return nullptr;
  }

  RefPtr<Promise> echoPromise = Promise::Create(mWindow->AsGlobal(), aRv);
  if (NS_WARN_IF(aRv.Failed())) {
    return nullptr;
  }
  ErrorResult rv;
  auto echoChild = GetEchoChild(rv);

  puts("[Navigator.cpp] sending Echo!");
  // Let's send aString to the parent!
  // We convert the string to UTF8
  echoChild->SendEcho(NS_ConvertUTF16toUTF8(aString))
      ->Then(
          GetMainThreadSerialEventTarget(), __func__,
          // Resolve lambda
          [echoPromise](const nsCString& returnedString) {
            puts("[Navigator.cpp] yay, we got a message back!");
            // Send the string back out a UTF16
            echoPromise->MaybeResolve(NS_ConvertUTF8toUTF16(returnedString));
          },
          // Reject lambda
          [echoPromise](mozilla::ipc::ResponseRejectReason&& aReason) {
            puts("[Navigator.cpp] boo, something went wrong!");
            echoPromise->MaybeReject(NS_ERROR_UNEXPECTED);
          });
  return echoPromise.forget();
}

Now, ./mach build; ./mach run.

Trying it out

In the developer console, you should now be able to do:

navigator.echo("this is a test").then(console.log);

If you see "this is a test", then all has gone well.

Done! see the final implementation of PEcho as a diff in Phabricator.