I create my own python scripts to scrape data from website and push into database.But i need to refresh the data on regular basis without spending any money on resources . I tried out multiple ways to trigger the job for data refresh .I feel the Heroku gives me the best options and experience to do it.
For free database i use Mongo DB free-tier.(there might be other free sandbox but i use Mongo DB for the easy of convience)
We need to have 3 python scripts created your github.
Above is the sample to connect MongoDB and to insert records.
The requirement.txt will contain all records that are dependency and run configuration will have python version in it.
Heroku Configuration:
Now you can create a pipeline and an application on Heroku platform and upload your code to it. At this point our project root folder consist with following three files.
You can select the Advance Scheduler add-on and provision to your application (there is a free option as below)
Then go to the Advance Scheduler home page. Using that page you can create your scheduled task. To Create that, click on ‘Create First Trigger’ button and it will prompt to enter the details for the task. You can give a name to identify the task and most importantly, the command to run the task. In this case our python script in the scheduler.py file can be initiate by python scheduler.py command.
Now just Execute trigger the job
