Have the fundamental understanding of Python Object-Oriented Programming
Apache Spark Tutorials from Udemy
Learning Gap and Lessons Learned
For a while, the python script did not run in the terminal. It ran fine in the interactive window, but I wanted it to run in the terminal as I saw in the tutorial. I tinkered with the environment variables and got a lot more confident setting them up. The issue was not the environment variables, however, it was the way the script was structured, namely, the absence of the following clause.
I found out about this by googling and typing this issue on Youtube. The resources that I referred to are as follows:
Notice in the link above that the environment variable to the path to
Needs to be set as well.
While I had trouble running the above program, I also looked into what is Anaconda and why I should be using it if ever. The Anaconda folder on my computer seemed quite convoluted with the libraries that I do not understand the uses for. The following video was handy in understanding just that. It basically is an all-in-one virtual environment and package manager. This will come handy as I will be working on multiple projects at the same time.
The tutorial also explains that I could have chosen to download miniconda as opposed to the full conda to avoid having a massive list of libraries that I do not use now.