How to run more than one unity instance on the linux server

[samby707]

#296

I had already tried training an agent using an unity build on linux server. Please refer to the above link on how I did it. The problem that I'm facing now is if I want to run parallel experiments, I can't. Suppose I'm training one unity agent in one GPU and try doing another training, it doesn't work. The observations stop and the agent stops moving when the other training is started. I changed the worker id in environment.py so that it works, but it doesn't. The unity launches but seems like there is no communication between unity and the python code. How to solve this?

[samby707]

Anybody. Please reply.please help.

[vincentpierre]

Hi,
I think you are the first person to try what you are doing, I am afraid that we do not have the expertise to help you. My current intuition is that both environments are trying to use the same display or compete for GPU resources. Could you try to make a simpler environment without camera inputs ? This would help us figure out if the visual observations are the problem. You could also try to send each environment to a different display but I am not sure how to achieve that.

[MarcoMeter]

@samby707
Here are some instructions for running a docker container to run ML-Agents. Though, I only tested it on a pretty simple environment, but at least headless. So if there is an issue with displays at all, you can give this a try.

DockerContainerInstructions.pdf

[samby707]

I created 2 separate virtual displays using xserver and attached them to two separate GPUs. I wrote the code such that the 2 python training codes are running on 2 complete separate GPUs. But the unity instances they are creating are by default taking the same GPU, indexed as zero. I think that might be the problem. Is there any way to program the unity environments such that they run on separate GPUs?

[samby707]

Another thing, I can't do the training without observations, because that is the only input to my model. Can you suggest a way such that without changing that, I can bypass the problem?

[MarcoMeter]

I got my first Docker Image up and running for ML-Agents version 0.2.1c.

You can try to use that as it supports headless builds of your Unity environment.
https://hub.docker.com/r/leckofunny/unity-ml-agents/

[gururise]

Very nice, could you update the docker image to use 0.2.1d? thanks!

[MarcoMeter]

In a couple of days, I'll create a docker repository which automatically makes new builds of the docker image to shrink down the effort (mostly due to low upload speed) of providing updates.

As of now, there are two ways to upgrade to 0.2.1d on your local docker container.

1: Upload the ml-agents's Python directory as zip to the Jupyter tree and extract it using the Troubleshooting_Utilities Notebook.

2: Mount a volume from your host into the container
nvidia-docker run -it -p 8888:8888 --volume /your_host_path/ml-agents-0.2.1d:/home/jovyan/ml-agents-0.2.1d --name ml-agents leckofunny/unity-ml-agents:latest

[samby707]

By default, the 2 unity instances get allocated to the same GPU. Is there any way that Unity can be instructed to use specific GPU?

[samby707]

please reply anybody.

[Lakrix]

is it possible to run headless mode while using visual observation?

[awjuliani]

Hi @Lakrix,

We are currently working on enabling a docker image that can collect visual observations. You can find the PR here: #547 .