Jobs

Members of a project in Hopsworks, can launch the following types of application via the Jobs service of a project:

  • Apache Spark
  • Apache Flink
  • Apache Beam with the Apache Flink runner.

If you are a beginner it is highly advisable to click on the Spark button at landing page under the available tours. It will guide you through launching your first Spark application!

Guided tours

To create a new job, click on the Jobs tab from the Project Menu and follow the steps below:

  • Step 1: Press the New Job button on the top left corner
  • Step 2: Give a name for you job
  • Step 3: Select one of the available job types
  • Step 4: Select the executable file of your job that you have uploaded earlier in a Dataset
  • Step 5 (Optional): Set the main class
  • Step 6: In the Configure and create tab you can manually specify the configuration you desire for your job and any additional dependencies and arbitrary Spark/Flink parameters.
  • Step 7: Click on the Create button
  • Step 8: Click on the Run button to launch your job, a dialog textbox will ask for any runtime arguments the job may require. If this job requires no arguments, the field can be left empty. The figure below shows the dialog.
Job runtime arguments

After creating a job by following the new job wizard, you can manage all jobs and their runs from the landing page of the Jobs service. The figure below shows a project with 6 jobs where 5 jobs are shown per page. When a job has run at least once, all past and current runs are then shown in the UI.

Jobs

Users can interact with the jobs in the following ways:

  1. Search jobs by using the Search text box
  2. Filter jobs by creation date
  3. Set the number of jobs to be displayed per page
  4. Run a job
  5. Stop a job, this stops all the ongoing runs of a job.
  6. Edit a job, for example change the Spark configuration parameters
  7. View Monitoring UI, with detailed Job information such as Spark UI, YARN, real-time logs and metrics
Job logs
  1. View a job’s details
Job real-time logs
  1. Make a copy of a job

10. Export a job, which prompts the user to download a json file. A job can then be imported by clicking on the New Job and then Import Job button.

Additionally, users click on a job and view additional information about their runs.

  1. Information about the run, such as location of log files and id.
  2. Stop a run
  3. Monitoring UI of this particular run
  4. View/Download stdout logs
  5. View/Download stderr logs
Job run logs