Observability with Momento in Node.js

Logging

Our goal for all of the Momento SDKs is to make sure that developers can direct Momento log output to the same destination that they are using for the rest of their application logs; therefore, we aim to be compatible with all of the popular logging frameworks for a given programming language.

There are many different logging libraries available for node.js. Some popular ones include:

• Pino
• Winston
• Bunyan
• Morgan

To ensure that Momento is compatible with all of these libraries (and more!), we provide a light-weight logging facade that you can use to wrap your favorite logging client. To use it, you simply need to implement the MomentoLoggerFactory and MomentoLogger interfaces:

export interface MomentoLogger {
  error(msg: string, ...args: unknown[]): void;
  warn(msg: string, ...args: unknown[]): void;
  info(msg: string, ...args: unknown[]): void;
  debug(msg: string, ...args: unknown[]): void;
  trace(msg: string, ...args: unknown[]): void;
}

export interface MomentoLoggerFactory {
  // eslint-disable-next-line @typescript-eslint/no-explicit-any
  getLogger(loggerName: string | any): MomentoLogger;
}

Your implementation will just be a thin wrapper around your logging library of choice. We provide a complete, working example implementation that uses the pino logger; you can find the source code for that here.

When you create an instance of your MomentoLoggerFactory, you can specify a specific logger level. Then you can get MomentoLogger instances from that factory that use the specified logger level.

// Setting the level to ERROR means you will see error messages but
// no trace, info, debug, or warning messages.
const errorLoggerFactory = new DefaultMomentoLoggerFactory(DefaultMomentoLoggerLevel.ERROR);
const errorLogger = errorLoggerFactory.getLogger('momento-error-logger');
errorLogger.error('error in the code!');

// Setting the level to DEBUG means you will see error, info, debug,
// and warning messages but no trace messages.
const debugLoggerFactory = new DefaultMomentoLoggerFactory(DefaultMomentoLoggerLevel.DEBUG);
const debugLogger = debugLoggerFactory.getLogger('momento-debug-logger');
debugLogger.debug('helpful debugging message');

Once you have defined your MomentoLoggerFactory and MomentoLogger, the last step is to configure your Momento client to use your preferred logger, like this:

return new CacheClient({
  configuration: Configurations.Laptop.v1(
    new PinoMomentoLoggerFactory({
      transport: {
        target: 'pino-pretty',
        options: {
          colorize: true,
        },
      },
    })
  ),
  credentialProvider: CredentialProvider.fromEnvironmentVariable({environmentVariableName: 'MOMENTO_API_KEY'}),
  defaultTtlSeconds: 60,
});

Then you should see log messages from Momento coming through your pino logging environment. In this case you should see log messages that look like this:

[1685649962168] INFO (CacheClient/4386 on mycomputer.local): Creating Momento CacheClient
[1685649962168] INFO (ControlClient/4386 on mycomputer.local): Creating cache: test-cache

Metrics

Metrics are measurements that provide quantitative information about system performance and behavior. They are numerical values captured and recorded over regular intervals, providing statistical data to aid in understanding the trends and patterns in a system.

For Momento, specifically, you might want to capture client-side metrics on the number of requests made, their duration, request or response size, or failure rates.

The most straightforward way to emit these metrics is to use one of the ExperimentalMetricsMiddleware classes. These classes emit metrics in JSON format:

(Momento: _ExperimentalMetricsLoggingMiddleware):
{
  "momento": {
    "numActiveRequestsAtStart": 1,
    "numActiveRequestsAtFinish": 1,
    "requestType": "_GetRequest",
    "status": 0,
    "startTime": 1697663118489,
    "requestBodyTime": 1697663118489,
    "endTime": 1697663118492,
    "duration": 3,
    "requestSize": 32,
    "responseSize": 2,
    "connectionID": "0"
  }
}

The metrics format is currently considered experimental; in a future release, once the format is considered stable, this class will be renamed to remove the Experimental prefix. The two middleware classes available are:

• ExperimentalMetricsLoggingMiddleware: will emit metrics to your chosen logger. WARNING: depending on your request volume, this middleware will produce a high volume of log output. If you are writing logs directly to local disk, be aware of disk usage and make sure you have log rotation / compression enabled via a tool such as logrotate.
• ExperimentalMetricsCsvMiddleware: will emit metrics to a CSV file. WARNING: enabling this middleware may have minor performance implications, so enable with caution. Depending on your request volume, the CSV file size may grow quickly, and neither sampling nor file compression / rotation are included at this time.

Log files and CSVs can be analyzed or shared with Momento to diagnose performance issues. You may also direct your logs to an AWS CloudWatch Log Group and create a CloudWatch dashboard to monitor your Momento requests; an example of launching a Momento metrics dashboard and optional example application is available in the Node.js SDK. The example Lambda and Fargate applications utilize the ExperimentalMetricsLoggingMiddleware class and CloudWatch metric filters to populate a dashboard like the one shown below.

Alternatively, you can capture client-side metrics using your own custom middleware that intercepts Momento gRPC calls and responses. Here is an example that uses OpenTelemetry and Prometheus to capture request count faceted by request type:

First, set up metrics in your application:

const resource = Resource.default();

const metricsExporter = new PrometheusExporter({}, () => {
  console.log('prometheus scrape endpoint: http://localhost:9464/metrics');
});

const meterProvider = new MeterProvider({
  resource: resource,
});

meterProvider.addMetricReader(metricsExporter);

metrics.setGlobalMeterProvider(meterProvider);

Then, create a middleware that captures the metric:

import {Middleware, MiddlewareRequestHandler} from '@gomomento/sdk';
import {metrics} from '@opentelemetry/api';
import {Counter} from '@opentelemetry/api/build/src/metrics/Metric';
import {
  MiddlewareMessage,
  MiddlewareMetadata,
  MiddlewareStatus,
} from '@gomomento/sdk/dist/src/config/middleware/middleware';

class ExampleMetricMiddlewareRequestHandler implements MiddlewareRequestHandler {
  private requestCounter: Counter;
  constructor(requestCounter: Counter) {
    this.requestCounter = requestCounter;
  }
  onRequestMetadata(metadata: MiddlewareMetadata): Promise<MiddlewareMetadata> {
    return Promise.resolve(metadata);
  }

  onRequestBody(request: MiddlewareMessage): Promise<MiddlewareMessage> {
    const requestType = request.constructor.name;
    this.requestCounter.add(1, {'request.type': requestType});
    return Promise.resolve(request);
  }

  onResponseMetadata(metadata: MiddlewareMetadata): Promise<MiddlewareMetadata> {
    return Promise.resolve(metadata);
  }

  onResponseBody(response: MiddlewareMessage | null): Promise<MiddlewareMessage | null> {
    return Promise.resolve(response);
  }

  onResponseStatus(status: MiddlewareStatus): Promise<MiddlewareStatus> {
    return Promise.resolve(status);
  }
}

/**
 * Basic middleware implementation that captures a request count metric. See experimental-metrics-csv-middleware.ts for
 * more comprehensive metrics, although be aware that that class is meant for troubleshooting and will eat disk space quickly.
 */
export class ExampleMetricMiddleware implements Middleware {
  private readonly requestCounter: Counter;
  constructor() {
    const meter = metrics.getMeter('metric-middleware-meter');

    this.requestCounter = meter.createCounter('momento_requests_counter', {
      description: 'Momento GRPC requests',
    });
  }

  onNewRequest(): MiddlewareRequestHandler {
    return new ExampleMetricMiddlewareRequestHandler(this.requestCounter);
  }
}

When you create the Momento CacheClient, add the middleware and the metric will be incremented with each request:

new CacheClient({
  configuration: Configurations.Laptop.v1().addMiddleware(new ExampleMetricMiddleware()),
  credentialProvider: CredentialProvider.fromEnvironmentVariable({
    environmentVariableName: 'MOMENTO_API_KEY',
  }),
  defaultTtlSeconds: 60,
});

Here is an example of the Grafana UI displaying a graph of get and set requests made against Momento:

Traces

Traces provide a detailed timeline of processes within an application, showing the relationship between different components and services involved in a specific request or operation. They allow developers to analyze the sequence and duration of these operations, facilitating a better understanding of how data flows through the system.

The Momento Node.js SDK uses gRPC internally to communicate with the Momento servers. OpenTelemetry provides a capability for auto-instrumenting all gRPC calls with traces. You don't need to add any middleware code to produce these traces, like you do for the metrics. Here is an example that automatically generates traces for these calls and exports them to Zipkin:

const resource = Resource.default();

const provider = new NodeTracerProvider({
  resource: resource,
});

const exporter = new ZipkinExporter();

provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

registerInstrumentations({
  instrumentations: [new GrpcInstrumentation()],
});

This needs to run before any Momento code.

Here is an example of the Zipkin UI displaying traces for a cache creation, a get, and a set:

If the performance of your application is impacted by trace generation, you should consider sampling them to cut down on the number of traces generated. You can do this with OpenTelemetry by setting two environment variables:

export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.1"

Setting these will ensure that only 10% of traces are created.