The default GlassFish installations has a JMX listening socket on port 8686 (for GlassFish v2ur2) or 8696 (for GlassFish v2.1-b60) that speaks some type of RMI protocol. To connect jconsole to this port, start jconsole and enter this URL:

    service:jmx:rmi:///jndi/rmi://www.example.com:8686/jmxrmi

and fill in "admin" as the username, plus the corresponding password. And change www.example.com to the GlassFish host, and change the port number to 8696 if you are running v2.1-something (I don't know when they changed the port number).

If you have firewalled this port, you can tunnel jmxrmi over ssh. You need to make ssh forward two ports: 8686 (or 8696) and a randomized port with a high port number. This unknown port number can be determined like this:

    # lsof -nPp 4711 | grep LISTEN

where you change "4711" to whatever the GlassFish pid is. The line below TCP *:8686 (LISTEN) will typically contain the second port number. If it doesn't work, try one of the other high port numbers. Then create the ssh tunnels, start jconsole, and enter localhost as the host in the jmxrmi URL.

Now you can track the memory and CPU load of GlassFish on your remote server. If the CPU load seems abnormally high, you can take a snapshot of all the threads by doing "jstack 4711" in a server shell. For some bizarre reason, jstack running as root refuses to dump stack traces of a GlassFish process that is running as some other user. So you need to do e.g. "sudo -u myglassfishuser jstack 4711" from your root shell.

[Footnote: GlassFish v2.1-b60a has moved back the rmi socket to port 8686 again]

A few "remind-myself" notes about various stumbling blocks that I encountered while writing my first Java EE code for GlassFish:

• You need $GLASSFISH_HOME/lib/javaee.jar in the classpath when compiling.
• The JMS Destination Resource must have the property Name=PhysicalQueue, or Name=PhysicalTopic for a topic.
• Resouce injection by annotations doesn't work for arbitrary classes or JSP sources. It only works for container managed components like EJBs, servlets, servlet filters, JSP tag handlers, application-client "main" classes,
• The server log is spammed with a multitude of these messaging exceptions:
  "com.sun.messaging.jmq.io.Packet cannot be cast to com.sun.messaging.jms.ra.DirectPacket"
  Follow the instructions here to have JMS run in a separate process. Note: this problem has been fixed in the GlassFish v2.1 beta.
  
• The web-app XML schema for "web.xml" must be version 2.5 for annotations to work.
• For injection to work in application clients, the injection variables must be static. And in order to inject EJBs the client must be packaged in an EAR file and declared in "application.xml".
• Injection variables cannot be static in servlets and filters.
• Persistent entity classes must be placed in jars under "lib" in the EAR file.
• Servlet and filter objects should not have EntityManagers as instance variables, since EntityManagers are not thread-safe. Instead, either use an EJB as an instance variable and let the EJB access the EntityManager, or use EntityManagerFactory to create a temporary EntityManager and manage the transactions yourself.
• Any webapp that requires login needs a "sun-web.xml" that declares <security-role-mapping>, otherwise no one is allowed access. Or if the webapp is packaged in an EAR file, the same mapping can be declared in "sun-application.xml". Also, the <realm-name> in "web.xml" cannot contain space characters.
• If you change the admin password, asadmin login is needed to update the stored credentials in ~/.asadminpass.
• Use @TableGenerator annotation for a portable way to generate primary keys for persistent @Entity objects. The generated keys are filled in by the first call to EntityManager.persist(), and the new key is committed by EntityManager.flush().
• Log Toplink-generated SQL statements by adding the following property under <persistence-unit> in "persistence.xml":
  <property name="toplink.logging.level" value="FINE"/>

The convenience of having instant access to an embedded production-quality database was one of the reasons why I found GlassFish interesting. However, even an embedded database must at times be accessible from other clients than the main client, for purposes such as installation, maintenance, debugging, etc. There are embedded databases such as Berkeley DB and SQLite that solve this with filesystem locks. Derby has another option: the embedded Derby can start a server thread that listens on a network socket. The GlassFish documentation recommends that you start a separate JVM process for the database server in order to run client applications that use network JDBC-urls, but this complicates the GlassFish service management in a way that isn't really necessary. After all, the GlassFish server process contains an embedded Derby which already has the capability to listen on sockets.

But I found no obvious way to turn on this feature in the GlassFish admin console. After wading through huge amounts of documentation, I finally found a simple solution: by setting the following system properties, GlassFish will start a Derby network server on localhost, which goes away when GlassFish is stopped.

    derby.system.home=/opt/glassfish/databases
    derby.drda.startNetworkServer=true
    derby.drda.host=localhost
    derby.drda.portNumber=1527

The above assumes that "/opt/glassfish" is the installation directory. The "asadmin start-database" command will then use "/opt/glassfish/databases" for network clients, so that's what we want for the embedded derby listener as well in order to be compatible. The last two properties are only needed if you want something other than the default values of localhost and 1527.

To turn this on in GlassFish, add these properties to the first domain configuration. Insert these lines somewhere among the <jvm-options> under <java-config> in "domains/domain1/config/domain.xml":

    <jvm-options>-Dderby.system.home=${com.sun.aas.installRoot}/databases</jvm-options>
    <jvm-options>-Dderby.drda.startNetworkServer=true</jvm-options>

First stop glassfish and any standalone derby server that might be running, then apply this patch and then start glassfish again, and voila! You can now use the Derby JDBC network client without starting a separate Derby server.

Addendum: a few weeks after writing this, I found out that in a freshly installed GlassFish, the Derby network server isn't started at boot, because the embedded-Derby driver doesn't get loaded until the first time it is used. One way to force this to happen is by accessing the built-in connection pool __TimerPool. This can be done from the command line:

    asadmin ping-connection-pool __TimerPool

Or you can deploy an application that creates a javax.ejb.Timer. This seems to have the permanent effect that __TimerPool is accessed automatically at boot time, causing the Derby embedded driver to load and start the network server.

[Update: I have since found that doing it like this is not a good approach. See "Embedded Derby on a socket, revisited"]