Add the script that can repeatedly scrapes the coursebook until finished

[mikehquan19]

Currently we have -resume flag for coursebook scraper that can resume scraping the coursebook until it scrapes everything. Check it out if you haven't yet.
To take a step further in automating the coursebook scraper in GCP, add the shell script that will repeatedly call the coursebook scraper with resume flag (in case it is terminted halfway) until every course is scraped successfully.

Make sure to put the limit on the number of attemps it can recall the resume.

[justinschwerdtfeger]

I have a few exams coming up, so I probably won't make any progress on this for a few weeks. If this needs to be done sooner than that, it might be best to assign to someone else.

[justinschwerdtfeger]

I'm thinking of implementing this into the Go code itself rather than a shell script because its more flexible. Some things are not worth restarting for, like missing environment variables.

This would require code changes for error handling, but I think we should do these regardless of whether we use a shell script or not to follow Go error handling best practices. Here are some of my suggestions:

• Don't use log.Fatal, log.Fatalf, or any other functions that call os.Exit. These end the program early and stops defers from being called, and there is no way to recover from os.Exit, making restarting the program from inside the Go code impossible.
• Panic should only be used when something really unexpected happens. For simple issues like missing an environment variable, the stack trace from panic isn't necessary, and can be confusing.
• Errors should be handled or returned instead of calling Fatal (or panic). Then from main, after all defers have been executed, we can simply log the error and end the program or even restart like what you suggest with a script.

Any additional thoughts?

[mikehquan19]

Yeah I agree with the part that we should refactor the Go code directly instead of adding any bash script is a better idea.

As for error-handling, it’d be nice if you could give me some examples directly from the scraper and show how it impacts current state of the scraper.

[justinschwerdtfeger]

I'd love to discuss this further at the next meeting