Skip to content

Commit 4eba310

Browse files
natemail-awsnatemail-aws
authored
Neuron SDK Release 2.23.0 (#1157)
1 parent 2e4feb8 commit 4eba310

File tree

155 files changed

+3963
-2228
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

155 files changed

+3963
-2228
lines changed

‎_ext/neuron_tag.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@
8383
'general/arch/neuron-hardware/neuron-core-v3',
8484
'/general/announcements/neuron2.x/announce-neuron-trn2',
8585
'/general/arch/neuron-features/logical-neuroncore-config',
86+
'/libraries/neuronx-distributed/context_parallelism_overview',
8687
'libraries/nxd-training/'
8788
]
8889

@@ -126,7 +127,8 @@
126127
'general/models/inference-inf2-trn1-samples',
127128
'general/models/training-trn1-samples',
128129
'general/arch/neuron-hardware/trainium',
129-
'general/arch/neuron-hardware/neuron-core-v2'
130+
'general/arch/neuron-hardware/neuron-core-v2',
131+
'/libraries/neuronx-distributed/context_parallelism_overview',
130132

131133
]
132134
clear_nc_v2_tag = [

‎conf.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,7 @@
195195

196196
#top_banner_message="<span>&#9888;</span><a class='reference internal' style='color:white;' href='https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/setup/setup-troubleshooting.html#gpg-key-update'> Neuron repository GPG key for Ubuntu installation has expired, see instructions how to update! </a>"
197197

198-
top_banner_message="Neuron 2.22.0 is released! check <a class='reference internal' style='color:white;' href='https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/index.html#latest-neuron-release'> What's New </a> and <a class='reference internal' style='color:white;' href='https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/announcements/index.html'> Announcements </a>"
198+
top_banner_message="Neuron 2.23.0 is released! check <a class='reference internal' style='color:white;' href='https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/index.html#latest-neuron-release'> What's New </a> and <a class='reference internal' style='color:white;' href='https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/announcements/index.html'> Announcements </a>"
199199

200200
html_theme = "sphinx_book_theme"
201201
html_theme_options = {

‎containers/tutorials/tutorial-oci-hook.rst‎

Lines changed: 72 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -13,15 +13,20 @@ devices to the container. In order to do that an environment variable,
1313
“AWS_NEURON_VISIBLE_DEVICES=ALL" can be used.
1414

1515
For the above environment variable to be used, the oci neuron hook has to be
16-
installed/configured.
16+
installed/configured.
17+
18+
.. important::
19+
20+
The Neuron OCI Hook is currently NOT supported with AL2023 on ECS. For workarounds,
21+
see the :ref:`oci-hook-workarounds` section below.
1722

1823
Install oci-add-hooks dependency on the Linux host
1924
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025

2126
.. important::
2227

23-
This step should run on the Linux host and not inside the container.
24-
28+
This step should run on the Linux host and not inside the container.
29+
2530

2631
`oci-add-hooks <https://github.com/awslabs/oci-add-hooks>`__ is an OCI
2732
runtime with the sole purpose of injecting OCI prestart, poststart, and
@@ -31,33 +36,33 @@ exposes Inferentia devices to the container.
3136

3237
.. code:: bash
3338
34-
sudo apt install -y golang && \
35-
export GOPATH=$HOME/go && \
36-
go get github.com/joeshaw/json-lossless && \
37-
cd /tmp/ && \
38-
git clone https://github.com/awslabs/oci-add-hooks && \
39-
cd /tmp/oci-add-hooks && \
40-
make build && \
41-
sudo cp /tmp/oci-add-hooks/oci-add-hooks /usr/local/bin/
42-
39+
sudo apt install -y golang && \
40+
export GOPATH=$HOME/go && \
41+
go get github.com/joeshaw/json-lossless && \
42+
cd /tmp/ && \
43+
git clone https://github.com/awslabs/oci-add-hooks && \
44+
cd /tmp/oci-add-hooks && \
45+
make build && \
46+
sudo cp /tmp/oci-add-hooks/oci-add-hooks /usr/local/bin/
47+
4348
Install the package that has oci hook software
4449
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4550

4651
.. important::
4752

48-
This step should run on the Linux host and not inside the container.
53+
This step should run on the Linux host and not inside the container.
4954

5055
For Inf1 install the following package
5156

5257
.. code:: bash
5358
54-
sudo apt-get install aws-neuron-runtime-base -y
59+
sudo apt-get install aws-neuron-runtime-base -y
5560
5661
For Trn1 install the following package
5762

5863
.. code:: bash
5964
60-
sudo apt-get install aws-neuronx-oci-hook -y
65+
sudo apt-get install aws-neuronx-oci-hook -y
6166
6267
For docker runtime setup Docker to use oci-neuron OCI runtime.
6368
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -68,8 +73,8 @@ point at oci-neuron OCI runtime. Install dockerIO:
6873

6974
.. code:: bash
7075
71-
sudo cp /opt/aws/neuron/share/docker-daemon.json /etc/docker/daemon.json
72-
sudo service docker restart
76+
sudo cp /opt/aws/neuron/share/docker-daemon.json /etc/docker/daemon.json
77+
sudo service docker restart
7378
7479
If the docker restart command fails, make sure to check if the docker
7580
systemd service is not masked. More information on this can be found
@@ -83,17 +88,17 @@ containerd to use the neuron oci hook
8388

8489
.. code:: bash
8590
86-
default_runtime_name = "neuron"
87-
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.neuron]
88-
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.neuron.options]
89-
BinaryName = "/opt/aws/neuron/bin/oci_neuron_hook_wrapper.sh"
91+
default_runtime_name = "neuron"
92+
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.neuron]
93+
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.neuron.options]
94+
BinaryName = "/opt/aws/neuron/bin/oci_neuron_hook_wrapper.sh"
9095
9196
9297
After that restart the containerd daemon
9398

9499
.. code:: bash
95100
96-
sudo systemctl restart containerd
101+
sudo systemctl restart containerd
97102
98103
For cri-o runtime, setup cri-o to use oci-neuron OCI runtime.
99104
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -103,12 +108,53 @@ cri-o to use the neuron oci hook
103108

104109
.. code:: bash
105110
106-
default_runtime_name = "neuron"
107-
[crio.runtime.runtimes.neuron]
108-
runtime_path = "/opt/aws/neuron/bin/oci_neuron_hook_wrapper.sh"
111+
default_runtime_name = "neuron"
112+
[crio.runtime.runtimes.neuron]
113+
runtime_path = "/opt/aws/neuron/bin/oci_neuron_hook_wrapper.sh"
109114
110115
After that restart the containerd daemon
111116

112117
.. code:: bash
113118
114-
sudo systemctl restart cri-o
119+
sudo systemctl restart cri-o
120+
121+
.. _oci-hook-workarounds:
122+
123+
OCI hook workarounds
124+
^^^^^^^^^^^^^^^^^^^^
125+
126+
**ECS (EC2)**
127+
128+
Add the following to your ECS task definition:
129+
130+
.. code:: json
131+
132+
"linuxParameters": {
133+
"devices": [
134+
{
135+
"containerPath": "/dev/neuron0",
136+
"hostPath": "/dev/neuron0",
137+
"permissions": [
138+
"read",
139+
"write"
140+
]
141+
},
142+
{
143+
"containerPath": "/dev/neuron1",
144+
"hostPath": "/dev/neuron1",
145+
"permissions": [
146+
"read",
147+
"write"
148+
]
149+
},
150+
...,
151+
],
152+
},
153+
154+
The linuxParameters parameter can be found under containerDefinition. More information can be found here:
155+
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_linuxparameters.
156+
Expose as many Neuron devices as needed, up to the max number of devices for the specified instance.
157+
For example, the trn1.32xlarge instance type contains 16 neuron devices, so the devices that can be exposed are
158+
/dev/neuron0, /dev/neuron1, up to /dev/neuron15.
159+
To see an example of an ECS task definition exposing Neuron devices,
160+
see https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-inference-task-def.html.

‎dlami/index.rst‎

Lines changed: 57 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -87,20 +87,20 @@ Virtual Environments pre-installed
8787
* - Neuron Framework/Libraries supported
8888
- Virtual Environment
8989

90-
* - PyTorch 2.5 Torch NeuronX, NxD Core
91-
- /opt/aws_neuronx_venv_pytorch_2_5
92-
93-
* - PyTorch 2.5 NxD Training, Torch NeuronX
94-
- /opt/aws_neuronx_venv_pytorch_2_5_nxd_training
95-
96-
* - PyTorch 2.5 NxD Inference, Torch NeuronX
97-
- /opt/aws_neuronx_venv_pytorch_2_5_nxd_inference
90+
* - PyTorch 2.6 Torch NeuronX, NxD Core
91+
- /opt/aws_neuronx_venv_pytorch_2_6
92+
93+
* - PyTorch 2.6 NxD Training, Torch NeuronX
94+
- /opt/aws_neuronx_venv_pytorch_2_6_nxd_training
95+
96+
* - PyTorch 2.6 NxD Inference, Torch NeuronX
97+
- /opt/aws_neuronx_venv_pytorch_2_6_nxd_inference
9898

99-
* - Transformers NeuronX (PyTorch 2.5)
100-
- /opt/aws_neuronx_venv_pytorch_2_5_transformers
99+
* - Transformers NeuronX (PyTorch 2.6)
100+
- /opt/aws_neuronx_venv_pytorch_2_6_transformers
101101

102-
* - JAX 0.4 NeuronX
103-
- /opt/aws_neuronx_venv_jax_0_4
102+
* - JAX 0.5 NeuronX
103+
- /opt/aws_neuronx_venv_jax_0_5
104104

105105
* - Tensorflow 2.10 NeuronX
106106
- /opt/aws_neuronx_venv_tensorflow_2_10
@@ -111,9 +111,12 @@ Virtual Environments pre-installed
111111
* - PyTorch 1.13 Neuron (Inf1)
112112
- /opt/aws_neuron_venv_pytorch_1_13_inf1
113113

114-
You can easily get started with the multi-framework DLAMI through AWS console by following this :ref:`setup guide <setup-ubuntu22-multi-framework-dlami>`. If you are looking to
115-
use the Neuron DLAMI in your cloud automation flows, Neuron also supports :ref:`SSM parameters <ssm-parameter-neuron-dlami>` to easily retrieve the latest DLAMI id.
116114

115+
Within the PyTorch 2.6 NxD Training virtual environment, we have included a setup script that installs required dependencies for the package. To run this script,
116+
activate the virtual environment and run ``setup_nxdt.sh`` and this will run :ref:`the setup steps here <nxdt_installation_guide>`.
117+
118+
You can easily get started with the multi-framework DLAMI through AWS console by following this :ref:`setup guide <setup-ubuntu22-multi-framework-dlami>`. If you are looking to
119+
use the Neuron DLAMI in your cloud automation flows, Neuron also supports :ref:`SSM parameters <ssm-parameter-neuron-dlami>` to easily retrieve the latest DLAMI id.
117120

118121
Neuron Single Framework DLAMI
119122
-----------------------------
@@ -135,15 +138,25 @@ Single Framework DLAMIs supported
135138
- Neuron Instances Supported
136139
- DLAMI Name
137140

138-
* - PyTorch 2.5
141+
* - PyTorch 2.6
142+
- Ubuntu 22.04
143+
- Inf2, Trn1, Trn1n, Trn2
144+
- Deep Learning AMI Neuron PyTorch 2.6 (Ubuntu 22.04)
145+
146+
* - PyTorch 2.6
147+
- Amazon Linux 2023
148+
- Inf2, Trn1, Trn1n, Trn2
149+
- Deep Learning AMI Neuron PyTorch 2.6 (Amazon Linux 2023)
150+
151+
* - JAX 0.5
139152
- Ubuntu 22.04
140153
- Inf2, Trn1, Trn1n, Trn2
141-
- Deep Learning AMI Neuron PyTorch 2.5 (Ubuntu 22.04)
154+
- Deep Learning AMI Neuron JAX 0.5 (Ubuntu 22.04)
142155

143-
* - PyTorch 2.5
156+
* - JAX 0.5
144157
- Amazon Linux 2023
145158
- Inf2, Trn1, Trn1n, Trn2
146-
- Deep Learning AMI Neuron PyTorch 2.5 (Amazon Linux 2023)
159+
- Deep Learning AMI Neuron JAX 0.5 (Amazon Linux 2023)
147160

148161
* - Tensorflow 2.10
149162
- Ubuntu 22.04
@@ -173,22 +186,26 @@ Virtual Environments pre-installed
173186
* - DLAMI Name
174187
- Neuron Libraries supported
175188
- Virtual Environment
189+
190+
* - Deep Learning AMI Neuron PyTorch 2.6 (Ubuntu 22.04, Amazon Linux 2023)
191+
- PyTorch 2.6 Torch NeuronX, NxD Core
192+
- /opt/aws_neuronx_venv_pytorch_2_6
176193

177-
* - Deep Learning AMI Neuron PyTorch 2.5 (Ubuntu 22.04, Amazon Linux 2023)
178-
- PyTorch 2.5 Torch NeuronX, NxD Core
179-
- /opt/aws_neuronx_venv_pytorch_2_5
194+
* - Deep Learning AMI Neuron PyTorch 2.6 (Ubuntu 22.04, Amazon Linux 2023)
195+
- PyTorch 2.6 NxD Training, Torch NeuronX
196+
- /opt/aws_neuronx_venv_pytorch_2_6_nxd_training
180197

181-
* - Deep Learning AMI Neuron PyTorch 2.5 (Ubuntu 22.04, Amazon Linux 2023)
182-
- PyTorch 2.5 NxD Training, Torch NeuronX
183-
- /opt/aws_neuronx_venv_pytorch_2_5_nxd_training
198+
* - Deep Learning AMI Neuron PyTorch 2.6 (Ubuntu 22.04, Amazon Linux 2023)
199+
- PyTorch 2.6 NxD Inference, Torch NeuronX
200+
- /opt/aws_neuronx_venv_pytorch_2_6_nxd_inference
184201

185-
* - Deep Learning AMI Neuron PyTorch 2.5 (Ubuntu 22.04, Amazon Linux 2023)
186-
- PyTorch 2.5 NxD Inference, Torch NeuronX
187-
- /opt/aws_neuronx_venv_pytorch_2_5_nxd_inference
202+
* - Deep Learning AMI Neuron PyTorch 2.6 (Ubuntu 22.04, Amazon Linux 2023)
203+
- Transformers NeuronX PyTorch 2.6
204+
- /opt/aws_neuronx_venv_pytorch_2_6_transformers
188205

189-
* - Deep Learning AMI Neuron PyTorch 2.5 (Ubuntu 22.04, Amazon Linux 2023)
190-
- Transformers NeuronX PyTorch 2.5
191-
- /opt/aws_neuronx_venv_pytorch_2_5_transformers
206+
* - Deep Learning AMI Neuron JAX 0.5 (Ubuntu 22.04, Amazon Linux 2023)
207+
- JAX NeuronX 0.5
208+
- /opt/aws_neuronx_venv_jax_0_5
192209

193210
* - Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 22.04)
194211
- Pytorch Neuron (Inf1)
@@ -279,11 +296,17 @@ SSM Parameter Prefix
279296
* - Deep Learning AMI Neuron (Amazon Linux 2023)
280297
- /aws/service/neuron/dlami/multi-framework/amazon-linux-2023
281298

282-
* - Deep Learning AMI Neuron PyTorch 2.5 (Ubuntu 22.04)
283-
- /aws/service/neuron/dlami/pytorch-2.5/ubuntu-22.04
299+
* - Deep Learning AMI Neuron PyTorch 2.6 (Ubuntu 22.04)
300+
- /aws/service/neuron/dlami/pytorch-2.6/ubuntu-22.04
284301

285-
* - Deep Learning AMI Neuron PyTorch 2.5 (Amazon Linux 2023)
286-
- /aws/service/neuron/dlami/pytorch-2.5/amazon-linux-2023
302+
* - Deep Learning AMI Neuron PyTorch 2.6 (Amazon Linux 2023)
303+
- /aws/service/neuron/dlami/pytorch-2.6/amazon-linux-2023
304+
305+
* - Deep Learning AMI Neuron JAX 0.5 (Ubuntu 22.04)
306+
- /aws/service/neuron/dlami/jax-0.5/ubuntu-22.04
307+
308+
* - Deep Learning AMI Neuron JAX 0.5 (Amazon Linux 2023)
309+
- /aws/service/neuron/dlami/jax-0.5/amazon-linux-2023
287310

288311
* - Deep Learning AMI Neuron PyTorch 1.13 Inf1 (Ubuntu 22.04)
289312
- /aws/service/neuron/dlami/pytorch-1.13-inf1/ubuntu-22.04

‎frameworks/jax/index.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,9 @@ a tested combination of the ``jax-neuronx``, ``jax``, ``jaxlib``, ``libneuronxla
1616
/frameworks/jax/setup/jax-setup
1717
/frameworks/jax/setup/jax-neuronx-known-issues
1818
/frameworks/jax/api-reference-guide/index
19+
/release-notes/jax-neuronx/index.rst
1920

2021
* :ref:`jax-neuron-setup`
2122
* :ref:`jax-neuron-known-issues`
2223
* :ref:`jax-neuronx-api-reference-guide`
24+
* :ref:`jax-neuronx-rn`

‎frameworks/tensorflow/tensorflow-neuronx/setup/tensorflow-update-u22.rst‎

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -14,18 +14,3 @@ If you already have a previous Neuron release installed, this section provide li
1414
.. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst
1515

1616
.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami --category=compiler_framework
17-
18-
19-
.. tab-item:: Tensorflow 2.9.3
20-
21-
.. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst
22-
23-
.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --neuron-version=2.10.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami --category=compiler_framework
24-
25-
26-
27-
.. tab-item:: Tensorflow 2.8.4
28-
29-
.. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst
30-
31-
.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --neuron-version=2.10.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami --category=compiler_framework

‎frameworks/torch/torch-neuronx/setup/pytorch-install-prev-al2023.rst‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,16 @@ This section will assist you to install previous Neuron releases.
1212

1313
.. tab-set::
1414

15+
.. tab-item:: Neuron 2.22.0
16+
17+
.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.5.1 --neuron-version=2.22.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami
18+
1519
.. tab-item:: Neuron 2.21.0
1620

1721
.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.5.1 --neuron-version=2.21.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami
1822

23+
1924
.. tab-item:: Neuron 2.20.0
2025

2126
.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.1.2 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami
2227

23-
.. tab-item:: Neuron 2.19.0
24-
25-
.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.1.2 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami

‎frameworks/torch/torch-neuronx/setup/pytorch-install-prev-u22.rst‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,16 @@ This section will assist you to install previous Neuron releases.
1212

1313
.. tab-set::
1414

15+
.. tab-item:: Neuron 2.22.0
16+
17+
.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.5.1 --neuron-version=2.22.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami
18+
1519
.. tab-item:: Neuron 2.21.0
1620

1721
.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.5.1 --neuron-version=2.21.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami
1822

23+
1924
.. tab-item:: Neuron 2.20.0
2025

2126
.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.1.2 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami
2227

23-
.. tab-item:: Neuron 2.19.0
24-
25-
.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.1.2 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami

0 commit comments

Comments
 (0)