MosaicML Composer integration API reference

API reference for the NeptuneLogger class of the MosaicML Composer integration.

TheNeptuneLoggerclass of the MosaicML Composer library captures metadata that is generated when training models with Composer.

You can use its.neptune_runproperty to access the associated Neptune run, and the.base_handlerproperty to assign custom metadata directly to the base namespace used for the metadata tracked by the logger.

NeptuneLogger

Creates a Neptune logger for tracking metadata during training with Composer.

Parameters

To keep your token secure, avoid placing it in the source code. Instead,save it as an environment variable.

Examples

If you have your Neptune credentials saved as environment variables, the following enables the Neptune logger with default settings:

from composer import Trainer
from composer.loggers import NeptuneLogger

neptune_logger = NeptuneLogger()

trainer = Trainer(
 ...,
 loggers=neptune_logger,
)

trainer.fit()

Set your Neptune API token and full project name to theNEPTUNE_API_TOKENandNEPTUNE_PROJECTenvironment variables, respectively.

export NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

export NEPTUNE_PROJECT="ml-team/classification"

export NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

export NEPTUNE_PROJECT="ml-team/classification"

setx NEPTUNE_API_TOKEN "h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

setx NEPTUNE_PROJECT "ml-team/classification"

You can also navigate toSettings→Edit the system environment variablesand add the variables there.

%env NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

%env NEPTUNE_PROJECT="ml-team/classification"

To find your credentials:

• API token : In the bottom-left corner of the Neptune app, expand your user menu and select Get your API token . If you need the token of a service account, go to the workspace or project settings and enter the Service accounts settings.
• Project name : Your full project name has the form workspace-name/project-name . You can copy it from the project menu ( → Details & privacy ).

If you're working in Google Colab, you can set your credentials with the os and getpass libraries:

import os
from getpass import getpass
os.environ["NEPTUNE_API_TOKEN"] = getpass("Enter your Neptune API token: ")
os.environ["NEPTUNE_PROJECT"] = "workspace-name/project-name"

You can also pass more options to the run created by the logger:

neptune_logger = NeptuneLogger(
 project="workspace-name/project-name", # (1)!
 upload_checkpoints=True,
 name="mellow-panda",
 description="Quick training run with Composer",
 tags=["training", "composer"],
 dependencies="infer",
)

1. The full project name. For example,"ml-team/classification".
2. You can copy the name from the project details ( → Details & privacy )
3. You can also find a pre-filled project string in Experiments → Create a new run .

The full project name. For example,"ml-team/classification".

To keep your token secure, avoid placing it in source code. Instead,save it as an environment variable.

If you leave it out, the value of theNEPTUNE_MODEenvironment variable is used. If that's not set, the defaultasyncis used.

List of source files to be uploaded. Must be list ofstror a singlestr. Uploaded sources are displayed in theSource codesection of the run.

IfNoneis passed, the Python file from which the run was created will be uploaded. When resuming a run, no file will be uploaded by default. Pass an empty list ([]) to upload no files.

Unix style pathname pattern expansion is supported. For example, you can pass".py"to upload all Python source files from the current directory. Paths of uploaded files are resolved relative to the calculated common root of all uploaded source files. For recursion lookup, use"**/.py"(for Python3.5and later). For details, see thegloblibrary.

IfNone, Neptune looks for a repository in the path of the script that is executed.

To specify a different location, set toGitRef(repository_path="path/to/repo").

To turn off Git tracking for the run, set toGitRef.DISABLEDorFalse.

Note:Instead of using this argument, you can use Neptune's default callback by setting theNEPTUNE_ENABLE_DEFAULT_ASYNC_LAG_CALLBACKenvironment variable toTRUE.

Note:Instead of using this argument, you can use Neptune's default callback by setting theNEPTUNE_ENABLE_DEFAULT_ASYNC_NO_PROGRESS_CALLBACKenvironment variable toTRUE.

.base_handler

Gets anamespace handlerfor the base logging namespace.

Use the handler to log extra metadata to the run and organize it under the base namespace (default:training).

You can operate on the handler like aRunobject: Access a path inside the handler and assign metadata to it with=or other logging methods from the Neptune client library. For some suggestions, seeEssential logging methods.

from composer.loggers import NeptuneLogger

neptune_logger = NeptuneLogger()
trainer = Trainer(loggers=neptune_logger, ...)
trainer.fit()

neptune_logger.base_handler["some_metric"] = 1
trainer.close()

Result

The value1is organized undertraining/some_metricin the run structure.

To learn more about namespace handlers, seeSetting a base namespace.

.neptune_run

Gets the NeptuneRunobject of theNeptuneLoggerinstance.

You can then access fields inside the run and use any logging methods from the Neptune client library. For some suggestions, seeEssential logging methods.

from composer.loggers import NeptuneLogger

neptune_logger = NeptuneLogger()
trainer = Trainer(loggers=neptune_logger, ...)
trainer.fit()

neptune_logger.neptune_run["your/metadata/structure"] = some_metadata
neptune_logger.neptune_run["some_file"].upload("path/to/some/file")

trainer.close()

Structurally, this will bypass the base namespace set for the logger.

By default, the metadata is logged in the namespace<run root>/training`, but the below code would logscoresdirectly at the run root and not undertraining`.

neptune_logger.neptune_run["scores"] = ...

To log under the base namespace instead, see.base_handlerproperty.

Related

NeptuneLoggerreference in theComposer documentation

This page is originally sourced from the legacy docs.