Pipeline 7.0 Release Notes

Filed under: Pipeline

March 24, 2017
Pipeline version 7.0 is just around the corner! Please read this post for full details about when the release will take place and what is new in this version!

Release Date

We plan to update the server from 7pm to 11pm PST on Tuesday March 28.

Important: Please do not submit any jobs via Pipeline that won’t complete before 6:30pm on Tuesday. Although once they make it to the grid, they will run without interruption, we cannot guarantee that you will be able to access that session in the Pipeline client when the server is back online.

After the update is complete, if you already have Pipeline installed you can launch your client and will be automatically prompted to update to the new version. Otherwise, the new version will be available for download on the products page.

What’s new in Pipeline 7.0?

Important: This version and future public release versions of Pipeline require Java 1.8 (latest) to run properly. If you don’t have it already installed, go to http://java.com to install it.

The newest version of Pipeline contains many new features and improvements to the client and server, such as to the Pipeline-IDA interface, job management abilities, user quotas and more, in addition to the usual slew of bug fixes. In this section we would like to give a brief overview of a few of the bigger changes. For a full textual list of the changes made, see the Release Notes.

1. Upgraded Database

The most important changes to software are many times the least visible to users. The LONI Pipeline Server is now running a faster and more reliable database behind the scenes. From an end users perspective, you may not notice this change unless you are running extremely large batches of jobs (which we encourage you to do) however this change will lead to overall less database and server downtime as well as improved performance.

Note: The distributed pipeline server (DPS) will continue to be packaged with the HSQL database.

2. Grid Usage Information

We have created a new Window for viewing detailed information about the grid and jobs. You can access it by clicking on the server statistics button location on the bottom right of the client, which will bring up a dialog containing a visual representation of every node in the grid as well as a bar graph breakdown of the slot distribution.

  • Clicking on any node in the grid will show information about that host including the jobs running on it. The grid can be sorted by various criteria
  • Clicking on any of the bars at the bottom of the window will bring up a list of the jobs pertaining to that bar. (e.x. The running jobs bar will bring up a list of all running jobs)
  • A small circle icon will appear inside any node that is running your jobs. You can click on the “View my Jobs” button to show all of your jobs in one window
  • The “Show Queue” option box can be used to highlight any node in the grid that the selected queue can submit jobs to.
  • Clicking on the “?” Icon in the top part of the dialog will bring up the “Pipeline and Grid Computing Diagram” which can serve as a good overview of what Pipeline and the Grid are and how they relate/communicate with each other

For developers: the grid information provided to the client is generated by a plugin that can be customized to work with any grid resource manager. For more information on how to make a plugin of your own, see the Pipeline Server Guide – Grid Stat Plugin API.

3. Collaborative Server Library

Users can request to make a variety of changes to the remote server library of modules and workflow by right clicking the server library navigator on the left side of the Pipeline client. For this Pipeline beta version, keep in mind that adding your workflows to this server’s library will not affect Cranium’s server library, however, we will transfer all changes when the beta testing phase is over.

      

4. Scripting in Pipeline

You can now create and edit local or remote* files from within the Pipeline client (* if they are on the same shared file system as the pipeline server). To begin creating or editing files, you can either click Tools > Script Editor from the top menu bar in the Pipeline window, or create a new module definition and click the “Edit Script” button from the Execution tab (see below screen shots).

      

5. Package Mapping Updates

Pipeline has included a package mapping system for quite some time now. But up until this release, information about the system has been hidden from users. Now users can select a package and version for their modules from a drop down list that is populated upon connecting to a server. The path of the package on the remote server is now displayed as well. This feature, along with the remote script editor will allow users to have more control and knowledge about remote programs they use.

For anyone unfamiliar with package mapping, it is a configurable system set up by the Pipeline server administrators that maps the name of a package to the root path of that package on the server. This allows you to specify just an executable name and Pipeline will know exactly on the server where that program is. As example, a user can write “reorient” as the module’s executable path and specify that it belongs to the FSL package version 4.1.0 and Pipeline will know to look in /usr/local/fsl-4.1.0_64bit/bin for that program.

6. For server admins: Job Management by module name

You can now set the maximum number of jobs of a particular “type” that can be running on the grid at any particular point in time. Please see the Pipeline Server Guide’s Configuration section called Non-user-based Job Management for more details about how to configure your server.