Univa Grid Engine is recognized as a high throughput Workload Management system, but as our users continue to grow their use of Univa Grid Engine they encounter applications and use cases that require massive job and task throughput for relatively short running work. Today our customers use Array Tasks to run these types of workload and for most of our customers Array Tasks work fine, however, if the runtime of the tasks is just a few seconds or even a second then the overhead of a traditional Workload Management system is considerable - it may even be longer than the runtime of the actual task. We recognized this and created Univa Short Jobs to address this type of Workload. With Univa Short jobs you can run thousands of tasks per second and even billions of tasks per day (approx 2.5 billion tasks). Short Jobs provides an order of magnitude improvement in task throughput compared to traditional 'batch' workload.
Short Jobs is very powerful, but it can also be quite complex compared to submitting a job with qsub, thus we created a collection of short videos that describes all of the major components of Short Jobs from installing the Short Jobs Add-on to using the tool with simple workflows and even complex workflows with dependencies.
Each day this week I will post a short jobs video describing a different aspect of the Add-on. Each video is short and no longer than 5 minutes. This first video describes how to install Short Jobs into a Univa Grid Engine Cluster. You may have to increase your YouTube video playback quality to see everything. Please have a look at the videos and comment, we love feedback.
Univa Short Jobs Video #1 - Introduction
Univa Short Jobs Video #2 - Architecture
Univa Short Jobs Video #3 - Web User Interface
Univa Short Jobs Video #4 - The qsub wrapper
Univa Short Jobs Video #5 - The submission API for Python
Univa Short Jobs Video #6 - The Submission API for Java
Univa Short Jobs Video #7 - Category Dependencies
Univa Short Jobs Video #8 - Task and Replication Dependencies
Univa Short Jobs Video #9 - The Execution API
Univa Short Jobs Video #10 - Fault Tolerance
Univa Short Jobs Video #11 - Black Hole Detection
Univa Short Jobs Video #12 - Checkpoint, Migration and Restart