How can I configure a custom logging formatter in Apache Beam Dataflow?

Below is some background on what’s happening with formatter configuration in YAML. Formatter objects (like those used with loggers) aren’t meant to be “serialized” and passed around in a dataflow. In a YAML‑based logging configuration the values (e.g. the format string, date format, etc.) are used to configure the formatter when you call the configuration routine (typically via dictConfig()). That’s why you can “override” log levels (simple values) but not pass an instantiated formatter object between processes or stages in a dataflow.

If you need to configure a custom formatter in your pipeline you have a couple of options:

• Use user‑defined instantiation. In your YAML you can provide a special key (the parenthesis key, “()”) with an absolute import path to your factory or callable. When dictConfig runs it will call that factory (passing the remaining keys as keyword arguments) to create your formatter. For example:

formatters:
  custom:
    (): my.package.customFormatterFactory
    bar: baz
    spam: 99.9
    answer: 42

This tells the configuration machinery to call:

my.package.customFormatterFactory(bar='baz', spam=99.9, answer=42)

• If you want to set additional attributes on your formatter object before it is returned, you can add a special “.” key with an attribute dict. For instance:

formatters:
  custom:
    (): my.package.customFormatterFactory
    bar: baz
    spam: 99.9
    answer: 42
    .:
      foo: bar
      baz: bozz

Keep in mind that logging configuration (and especially custom objects like formatters) is intended to be set up once at initialization rather than passed around as already‑instantiated objects in a dataflow. If your pipeline requires passing configuration changes dynamically, consider instead sending or re‑computing the parameters in a way that your configuration loader (or custom factory) can re‑create the formatter at the appropriate point.

In summary, there isn’t a built‑in “serialization” option for formatters similar to log level overrides. Instead, use the factory mechanism in your YAML so that the formatter is properly instantiated as needed from your configuration data.