Kedro integration API reference

API reference for the Kedro-Neptune plugin.

You can use the Kedro-Neptune plugin to log metadata about the Kedro pipeline execution to Neptune.

Configuration

Logging configuration

You can configure where and how Kedro pipeline metadata is logged by editing the parameters in theconf/base/neptune.ymlfile.

Options	Parameter	Default	Description
neptune	project	$NEPTUNE_PROJECT	Name of the Neptune project where metadata is stored.If you haven't set the$NEPTUNE_PROJECTenvironment variable, you can replace it with the full name of your Neptune project (workspace-name/project-name).
	base_namespace	kedro	Base namespace (folder) where your metadata will be logged.
	dependencies		Tracks environment requirements. If you passinferto this argument, Neptune logs dependencies installed in the current environment. You can also pass a path to your dependency file directly. If left empty, no dependency file is uploaded.
	enabled	true	If Neptune logging is enabled.
	upload_source_files	- '/.py'- conf/base/.yml	List of files you want to upload to Neptune.

If you haven't set the$NEPTUNE_PROJECTenvironment variable, you can replace it with the full name of your Neptune project (workspace-name/project-name).

- '**/*.py'

- conf/base/*.yml

Example

neptune:
#GLOBAL CONFIG
 project: $NEPTUNE_PROJECT
 base_namespace: kedro
 dependencies: infer
 enabled: true

#LOGGING
 upload_source_files:
 - '**/*.py'
 - conf/base/*.yml

Configuring Neptune API token

You can configure where the Kedro-Neptune plugin will look for your Neptune API token in theconf/local/credentials_neptune.ymlfile:

neptune:
 api_token: $NEPTUNE_API_TOKEN

You can:

leave it empty, in which case Neptune will look for your token in the$NEPTUNE_API_TOKENenvironment variable. ImportantThere must be a dollar sign ($) before the variable name.
pass a different environment variable, such as$MY_SPECIAL_NEPTUNE_API_TOKEN_VARIABLE.
pass your token as a string, such as eyJhcGlfYWRk...Imh0dHBzOi8v (not recommended).

leave it empty, in which case Neptune will look for your token in the$NEPTUNE_API_TOKENenvironment variable.

Important

There must be a dollar sign ($) before the variable name.

pass a different environment variable, such as$MY_SPECIAL_NEPTUNE_API_TOKEN_VARIABLE.

In the bottom-left corner of the Neptune app, open the user menu and selectGet your API token.

You can copy your token from the dialog that opens. It's very long – make sure to copy and paste it in full!

kedro neptune

CLIcommand for initializing the Kedro-Neptune plugin.

You can monitor, visualize, and compare your pipelines and node outputs in the Neptune app.

Note

You can only use thekedro neptunecommand in the Kedro project folder.

Command syntax:kedro neptune [OPTIONS] COMMAND [ARGS...]

Options	Description
--help	ShowsCLIusage: options, commands and their arguments.

kedro neptune init

Initializes the plugin and configures Neptune credentials and logging options.

After executing this command, whenever you executekedro run, you will log:

Parameters
Pipeline execution configuration ( run_params )
Metadata about Kedro Dataset
Hardware consumption and node execution time
Configuration files from the conf/base directory
Full Kedro run command
Any additional metadata like metrics, charts, or images that you logged from inside of your node functions.

Parameters

Name	Type	Default	Description
api-token	str, optional	$NEPTUNE_API_TOKEN	User's Neptune API token, or the name of the environment variable where it is stored. You can find your API token in the user menu, in the bottom-left corner of the app.
project	str, optional	$NEPTUNE_PROJECT	Neptune project name, or the name of the environment variable where it is stored. You can find and copy your project name from the project settings→Details & privacy.
base-namespace	str, optional	experiment	Namespace in Neptune where all the Kedro-related metadata is logged.
config	str, optional	base	Name of the subdirectory inside of the Kedroconfdirectory for configuration and catalog files.
dependencies	str, optional	None	Tracks environment requirements. If you passinferto this argument, Neptunelogs dependenciesinstalled in the current environment. You can also pass a path to your dependency file directly. If left empty, no dependency file is uploaded.

Examples

kedro neptune init

kedro neptune init \
 --api-token $NEPTUNE_API_TOKEN \
 --dependencies infer

NeptuneFileDataset

NeptuneFileDatasetis a Kedro Data Set that lets you log files to Neptune.

It can be any file on the POSIX compatible filesystem.

To log it, you need to define theNeptuneFileDatasetin any Kedro catalog, includingcatalog.yml.

Parameters

Name	Type	Description
filepath	str	In POSIX format, filepath to a text file prefixed with a protocol, such ass3://. Same as forKedro TextDataset.
credentials	dict, optional	Credentials required to access the underlying filesystem. Same as forKedro TextDataset.
fs_args	dict, optional	Extra arguments to pass to underlying filesystem class constructor. Same as forKedro TextDataset.

Examples

predictions:
 type: kedro.extras.datasets.json.JSONDataset
 filepath: data/07_model_output/predictions.json

predictions@neptune:
 type: kedro_neptune.NeptuneFileDataset
 filepath: data/07_model_output/predictions.json