Lifecyle

You can easily access and consume Kernels from various interfaces, including JupyterLab, the CLI or VS Code.

info

Read below to learn more about the available Kernel Actions.

• Launch.
• List.
• Assign.
• Execute Code.
• Interrupt.
• Restart.
• Pause.
• Resume.
• Transfer State.
• Terminate.

Launch a Kernel

Launching a Kernel is the first step to unlock the power of Datalayer.

SaaS

You can launch Kernels from the "Kernels" tab.

You will be prompted to select a few options:

• An Environment, this can be a Platform or a User Environment.

• For Remote Kernels, a Time Reservation that defines the maximum execution time for the Kernel. After this time, the Kernel will be automatically terminated to prevent overconsumption of Credits.

• You can choose whether the Kernel will have access to persistent persistent storage, allowing data to be retained between Kernel Sessions. Note that Kernels with persistent storage may take longer to start (approximately 15 seconds instead of 3 seconds). Persistent storage is available under the ~/persistent directory on the Remote Kernel.

• You can also open an example Notebook when the Kernel starts. This feature provides a ready-to-use notebook, helping you quickly familiarize yourself with the environment and start working right away.

JupyterLab

You can launch Kernels from the "Home" page of the extension.

You will be prompted to select a few options:

• An Environment, this can be a Platform or a User Environment.

• For Remote Kernels, a Time Reservation that defines the maximum execution time for the Kernel. After this time, the Kernel will be automatically terminated to prevent overconsumption of Credits.

• You can choose whether the Kernel will have access to persistent storage Content, allowing data to be retained between Kernel Sessions. Note that Kernels with persistent storage may take longer to start (approximately 15 seconds instead of 3 seconds). Persistent storage is available under the ~/persistent directory on the Remote Kernel.

• You can also open an example Notebook when the Kernel starts. This feature provides a ready-to-use notebook, helping you quickly familiarize yourself with the environment and start working right away.

CLI

Use datalayer kernels create ENV_ID --given-name GIVEN_NAME --credits-limit CREDITS_LIMIT.

datalayer kernels create python-simple-env --given-name k1 --credits-limit 3

                                            Jupyter Kernels                                              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Kernel ID                          ┃ Kernel Name ┃ Environment       ┃ Expired At                      ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ jupyter-01j2m0zt2kaheyy3yzn021g032 │ k1          │ python-simple-env │ 2024-08-10T17:29:19.355000+02:… │
└────────────────────────────────────┴─────────────┴───────────────────┴─────────────────────────────────┘

List the Kernels

SaaS

You can view the list of running Kernels in the "Kernels" tab.

JupyterLab

You can view the list of running Kernels in the "Home" page of the extension.

CLI

Use datalayer kernels list to list all the running Kernels.

datalayer kernels list

                                             Jupyter Kernels                                              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Kernel ID                          ┃ Kernel Name ┃ Environment       ┃ Expired At                      ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ jupyter-01j2m0zt2kaheyy3yzn021g032 │ k1          │ python-simple-env │ 2024-08-10T17:29:19.355000+02:… │
│ jupyter-01j2m1kc0zgq3kjcne4p5avhd8 │ k2          │ python-simple-env │ 2024-07-27T06:52:44.704000+02:… │
└────────────────────────────────────┴─────────────┴───────────────────┴─────────────────────────────────┘

Assign a Kernel

Once your Kernel is up and running, you can assign it to a Notebook or a specific cell.

Assign a Kernel to a Notebook

SaaS

warning

🚧 SaaS is Work in Progress

JupyterLab

You can change the kernel of any Notebook to a Remote Kernel by using the Kernel Picker. The Kernel Picker is located in the top-right corner of the Notebook interface.

You can also launch a new Remote Kernel directly from the Kernel Picker by selecting an environment in the 'Assign a new Kernel' section. You'll have the option to configure the Kernel settings, such as Time Reservation.

VS Code

In the SaaS, go to the action menu of the kernel and select the Copy link for IDE action.

This will copy in your clipboard a link to use in VS Code.

Go to your VS Code Jupyter Notebook, click on the Select Kernel button.

Choose Existing Jupyter Server, next paste the link you have previously copied, click Enter and choose the Python3 kernel.

success

That's it, your VS Code Notebook is connected to your Remote Kernel 👍.

info

You can connect to a Kernel from other IDEs like PyCharm as explained on the official PyCharm documentation.

CLI

warning

🚧 CLI is Work in Progress

Assign a Kernel to a Cell

Datalayer allows you to selectively run specific Notebook cells on Remote Kernels. This hybrid approach optimizes both performance and cost by using Remote resources only when necessary.

Learn more about this feature in our Blog Post.

SaaS

warning

🚧 SaaS is Work in Progress

JupyterLab

To assign a cell to a Remote Kernel, simply use the cell menu as shown below.

CLI

warning

🚧 CLI is Work in Progress

Execute Code

Execute a Notebook

SaaS

warning

🚧 SaaS is Work in Progress

JupyterLab

To execute a notebook on the Remote Kernel, you can use the JupyterLab interface.

VS Code

To execute a notebook on the Remote Kernel, you can use the VS Code interface.

CLI

Use datalayer kernels exec NOTEBOOK_PATH with the following command options:

• --kernel-id - Kernel ID of the Remote Kernel to use.
• --raise - Stop the execution one an exception occurred. By default, all cells of the notebook will be executed even if one cell fails.
• --timeout seconds
• --verbose - Display all cell outputs. By default, only cell outputs targeting the error stream will be displayed.

datalayer kernels exec hello.ipynb --kernel jupyter-01j2krcmskpye0sn0nv9a7afhp

Execute a File

SaaS

warning

🚧 SaaS is Work in Progress

JupyterLab

warning

🚧 JupyterLab is Work in Progress

CLI

Use datalayer kernels exec FILE_PATH with the following command options:

• --kernel-id - Kernel ID of the Remote Kernel to use.
• --raise - Stop the execution one an exception occurred. By default, all cells of the notebook will be executed even if one cell fails.
• --timeout seconds
• --verbose - Display all cell outputs. By default, only cell outputs targeting the error stream will be displayed.

cat > hello.py <<EOF
print('👋 Hello from Kernel')
EOF
datalayer kernels exec hello.py --kernel jupyter-01j2krcmskpye0sn0nv9a7afhp
rm hello.py

Execute Code in a Console

SaaS

warning

🚧 SaaS is Work in Progress

JupyterLab

warning

🚧 JupyterLab is Work in Progress

CLI

Use datalayer kernels console --kernel KERNEL_ID

datalayer kernels console --kernel jupyter-01j2krcmskpye0sn0nv9a7afhp

[KernelConsoleApp] Jupyter Kernels - Version 1.1.9 - Connected as eric on https://prod1.datalayer.run
[KernelConsoleApp] KernelManager using existing kernel server jupyter-01j2krbpy5nh72xcgc8r7jwjh7 expiring at 2024-08-10T17:16:58.139000+02:00

JupyterKernels console 1.1.9

Python 3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:36:13) [GCC 12.3.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.22.2 -- An enhanced Interactive Python. Type '?' for help.

In [1]: print('👋 Hello from Kernel')
👋 Hello from Kernel

In [2]: 

The Kernel ID is optional. If not provided, it will take the first Remote Kernel found.

warning

To keep the Kernel alive when you exit the console, use CTRL-D or quit(keep_kernel=True).

Interrupt a Kernel

You can interrupt a Kernel by clicking the Interrupt icon. This action will stop the Kernel's current execution.

This is useful when you want to stop a long-running computation or if the Kernel is unresponsive.

SaaS

JupyterLab

CLI

warning

🚧 CLI is Work in Progress

Restart a Kernel

You can restart a Kernel by clicking the Restart icon. This action will restart the Kernel, clearing all variables and outputs.

This is useful when you want to start fresh without creating a new Kernel.

SaaS

JupyterLab

CLI

warning

🚧 CLI is Work in Progress

Pause a Kernel

You can pause a Kernel by clicking the Pause icon. This action will pause the Kernel, allowing you to resume it later. This is useful when you want to temporarily stop the Kernel, keep its state, and resume it later.

With this feature, you can save credits by pausing the Kernel when you are not actively using it and resuming it when needed without having to rerun all the code that was needed to reach the current state.

note

When a Kernel is paused, a Kernel Snapshot is created.

SaaS

JupyterLab

CLI

warning

🚧 CLI is Work in Progress

warning

There are a few limitations to be aware of:

1. Object Size Limitations: Extremely large objects, such as massive datasets or memory-heavy models, may exceed persistence limits. Consider external storage options, like cloud storage or databases, for these cases.
2. Picklable Objects Only: The Kernel state is serialized using Python's pickle module. While common types (e.g., lists, dictionaries, NumPy arrays) are compatible, some objects (e.g., open file handles, certain class instances) may not be. Review the types of objects in your notebook to ensure compatibility.

Resume a Kernel

To Resume a Kernel, click the Resume icon of the Snapshot.

SaaS

JupyterLab

CLI

warning

🚧 CLI is Work in Progress

You can see the pause and resume feature in action below.

For details on snapshots features, see Snapshots.

Transfer a Kernel State

When you switch to a Remote Kernel, you have the option to transfer the variables from the current kernel to the new one.

This feature is useful when you want to keep the state of your variables when switching kernels. This allows to save time and resources by avoiding re-running the code to reach the current state.

SaaS

warning

🚧 SaaS is Work in Progress

JupyterLab

CLI

warning

🚧 CLI is Work in Progress

warning

Variable transfers are currently limited to 7 MB. If you have large variables, consider saving them in the persistent storage. For more information, see Persistent Storage. Additionally, only pickable objects can be transferred.

You can also transfer variables between cells running on different kernels. This feature is particularly useful when you want to use variables you defined in previous cells.

SaaS

warning

🚧 SaaS is Work in Progress

JupyterLab

CLI

warning

🚧 CLI is Work in Progress

Terminate a Kernel

You can terminate a Kernel by clicking the trash icon. This action will terminate the Kernel and remove it from the list of running Kernels. This is useful when a Kernel is no longer needed.

SaaS

JupyterLab

CLI

Use datalayer kernels terminate KERNEL_ID.

datalayer kernels terminate jupyter-01j2ks4xzbr0vwe8zfzcv5ksm2

[KernelTerminateApp] Jupyter Kernels - Version 1.1.9 - Connected as eric on https://prod1.datalayer.run
[KernelTerminateApp] Kernel 'jupyter-01j2ks4xzbr0vwe8zfzcv5ksm2' deleted.

tip

This will stop the credits consumption for this Kernel.