Kernels

Datalayer introduces Remote Kernels to seamlessly scale your Data Science and AI workflows.

info

A Jupyter Kernel is the computational engine that executes the code in your Jupyter Notebooks or Python scripts. It operates independently of the Notebook or script itself, allowing you to run your code remotely across various systems.

Types of Kernels

Kernels can be hosted in your browser, on your local Jupyter Server, or remotely.

• Browser Kernels: These kernels run as web workers directly within your browser, enabling seamless interaction without the need for additional software installations.
• Local Kernels: In this traditional setup, a Jupyter Server runs locally, and each execution spawns a local process to handle computations.
• Remote Kernels: These kernels run on a separate machine from your local device, leveraging potentially more powerful CPU, RAM, and GPU resources.

You can easily access and consume kernels from various interfaces, including JupyterLab, the CLI, or VS Code.

Launching Remote Kernels

You can launch Remote Kernels from the "Kernels" tab in Datalayer or via the "Home" page of the JupyterLab extension (see JupyterLab for installation instructions).

When launching a Remote Kernel, you'll be prompted to configure a few options:

Environment

You will be prompted to select an environment (see Environments for more details).

Time Reservation

This defines the maximum execution time for the kernel. After this time, the kernel will be automatically terminated to prevent overconsumption of credits (for more information, see Credits).

Persistent Storage

You can choose whether the kernel will have access to persistent storage, allowing data to be retained between kernels sessions. Note that kernels with persistent storage may take longer to start (approximately 15 seconds instead of 3 seconds).

Persistent storage is available under the persistent directory on the Remote Kernel (see Content for more details).

note

More options to store data and share it between Remote and Local Environments will be available soon, providing greater ease and flexibility in managing and accessing your data.

Example Notebook

You can enable the option to automatically open a preloaded example notebook when the kernel starts. This feature is available only for JupyterLab and provides a ready-to-use notebook, helping you quickly familiarize yourself with the environment and start working right away.

Kernels Management

You can view and manage running kernels under the "Kernels" tab, where you can delete, interrupt, restart or pause them as needed.

Delete Kernel

You can delete a kernel by clicking the trash icon. This action will terminate the kernel and remove it from the list of running kernels. This is useful when a kernel is no longer needed. This will stop the credits consumption for this kernel.

Interrupt Kernel

You can interrupt a kernel by clicking the Interrupt icon. This action will stop the kernel's current execution. This is useful when you want to stop a long-running computation or if the kernel is unresponsive.

Restart Kernel

You can restart a kernel by clicking the Restart icon. This action will restart the kernel, clearing all variables and outputs. This is useful when you want to start fresh without creating a new kernel.

Pause Kernel

You can pause a kernel by clicking the Pause icon. This action will pause the kernel, allowing you to resume it later. This is useful when you want to temporarily stop the kernel, keep its state, and resume it later. With this feature, you can save credits by pausing the kernel when you are not actively using it and resuming it when needed without having to rerun all the code that was needed to reach the current state.

When a kernel is paused, a Kernel Snapshot is created.

note

There are a few limitations to be aware of:

1. Object Size Limitations: Extremely large objects, such as massive datasets or memory-heavy models, may exceed persistence limits. Consider external storage options, like cloud storage or databases, for these cases.
2. Picklable Objects Only: The Kernel state is serialized using Python's pickle module. While common types (e.g., lists, dictionaries, NumPy arrays) are compatible, some objects (e.g., open file handles, certain class instances) may not be. Review the types of objects in your notebook to ensure compatibility.

Resume Kernel

To resume the kernel, click the Resume icon of the Snapshot.

You can see the pause and resume feature in action below: