Airflow integration API reference

API reference for the Neptune-Airflow integration.

You can use the Neptune integration with Apache Airflow to log metadata generated during DAG runs.

NeptuneLogger

Creates a Neptune logger instance for tracking metadata during a DAG run.

Parameters

Name	Type	Default	Description
api_token	str, optional	None	User's Neptune API token. If None, the value of theNEPTUNE_API_TOKENenvironment variable is used.To keep your token secure, avoid placing it in the source code. Instead,save it as an environment variable.
project	str, optional	None	Name of a project in the formworkspace-name/project-name. If None, the value of theNEPTUNE_PROJECTenvironment variable is used.
**neptune_run_kwargs	str, optional	-	Additional keyword arguments to be passed directly to theinit_run()function, such asdescriptionandtags.Note: Thecustom_run_idparameter is reserved in this integration. It's automatically generated based on the DAG ID.

To keep your token secure, avoid placing it in the source code. Instead,save it as an environment variable.

Note: Thecustom_run_idparameter is reserved in this integration. It's automatically generated based on the DAG ID.

Examples

If you have yourNeptune credentials saved as environment variables, the following creates a Neptune logger with default settings:

from neptune_airflow import NeptuneLogger

with DAG(
 ...
) as dag:

 def task(**context):
 neptune_logger = NeptuneLogger()

get_run_from_context()

Gets the run from the current task so it can be used for logging metadata within the task.

Parameters

Name	Type	Default	Description
context	dict	-	Apache Airflow Context received by the task.
log_context	bool	False	Whether to log the contents of the Context keyword arguments.

Returns

Runobject that can be used for logging.

Examples

with DAG(
 ...
) as dag:
 def task_1(**context):
 run = get_run_from_context(context)
 run["some_metric"] = 0.99

get_task_handler_from_context()

Gets anamespace handlernamed by the ID of the current task.

You can use the handler for logging metadata within the task, using the same logging methods as for runs. The metadata will be organized under therun["task_id"]namespace inside the run.

Parameters

Name	Type	Default	Description
context	dict	-	Apache Airflow Context received by the task.
log_context	bool	False	Whether to log the contents of the Context keyword arguments.

Returns

Handlerobject that can be used for logging.

Examples

with DAG(
 ...
) as dag:
 def task_1(**context):
 task_handler = get_task_handler_from_context(context)
 task_handler["some_metric"] = 0.99