swarm: adds handling GET robots.txt #154

[aquiladev]

No description provided.

[nonsense]

@aquiladev thank you for your contribution. However I am not sure we want to handle robots.txt in Swarm - generally this is done on the gateway layer (nginx for example).

Also it should be robots.txt, not robot.txt.

@gbalint what do you think?

[aquiladev]

@nonsense thx for reply. I do agree that it should be in gateway, but I don't know where it is. Could you point me, I'll check it.

[nonsense]

@aquiladev the swarm-gateways.net gateway is maintained by the Swarm team and is currently in a private repo. Any user is free to start their own gateway and expose it on the internet and decide for themselves how they want to handle the robots.txt.

For example there is a gateways located at eth.show which might want to handle this differently to swarm-gateways.net.

[nonsense]

Thanks, @aquiladev! We discussed this in the team, and decided that it makes sense to support this default configuration. Then in case users want a different robots.txt for their gateway, they can override Swarm.