Follow-on PR #911: Spark documentation rework #1098
Conversation
Some changes to the Spark documentation for local and standalone use cases with the following drivers * Simplify some of them (removing options, etc.) * Use the same code as much as possible in each example to be consistent (only kept R different from the others) * Add Sparklyr as an option for R * Add some notes about prerequisites (same version of Python, R installed on workers)
romainx
added
the
tag:Documentation
There was a problem hiding this comment.
Thanks for doing this cleanup, @romainx. The restructuring and simplification look good to me.
Were you able to prove that all the samples are functional by running them in the latest image?
Separate from this PR, we should consider putting all these snippets into tests for the all-spark-notebook image.
* Test added for all kernels * Same examples as provided in the documentation (`specifics.md`) * Used the same use case for all examples: sum of the first 100 whole numbers Note: I've not automatically tested `local_sparklyr.ipynb` since it creates by default the `metastore_db` dir and the `derby.log` file in the working directory. Since I mount it in `RO` it's not working. I'm struggling to set it elsewhere...
|
Thank you @parente Yes all the examples are working. I've reworked them a bit to use the same use case for all: "sum of the first 100 whole numbers".
Note: I've not automatically tested I let you decide but I think we could merge it and fix this after since it's not mandatory and since all kernels are tested ( Best |
|
It's great that you got most of the tests working already. Let's merge it! 🎉 |
2 participants
Some changes to the Spark documentation following the PR #911
for local and standalone use cases with the following drivers
Best