Singularity 0.16.1

Changes in 0.16.1

This is a bugfix release

Check out the 0.16.1 milestone to see new features / bugfixes in detail.

• 1580 - Fix mapping of docker fields in 'new deploy' ui

Singularity 0.16.0

Changes in 0.16.0

Check out the 0.16.0 milestone to see new features / bugfixes in detail.

Important

The next release of Singularity (0.17.0) will contain a bump to mesos 1.1.2. Any critical bug fixes will be backported to 0.16 for a short period. See #1571 for more details on the upcoming upgrade

New Endpoints

1559 Added two additional endpoints to more easily track the lifecyle of a task. Both endpoints return a SingularityTaskState object with basic details about the current state of the task

• /track/task/{taskId} - for tracking active tasks which have already been assigned an id
• /track/run/{requestId}/{runId} - For tracking tasks by runId (e.g. an ON_DEMAND task). This endpoint can also search pending tasks

Improvements

• 1560 - Separate timeouts for health check and task running
• 1568 - Better bash escaping in docker runner script
• 1515 - Add flag to trigger run of scheduled task on deploy
• 1561 - Combine offers to schedule tasks more efficiently
• 1562 - Retry failed client requests in SingularityClient
• 1538 - Allow immediate runs in pending queue with deploy
• 1482 - Better task balancing
• 1557 - Immediate uploads on executor teardown
• 1555 - Tailer improvements for tail_of_finished logs
• 1549 - Baragon 0.5.0
• 1526 - Add skip lb removal flag to DeleteRequestRequest
• 1534 - Upgrade docker client
• 1533 - Add mark as active/inactive endpoints to client
• 1524 - Easy endpoint to check if user is authorized for request

Bug Fixes

• 1572 - Fix bug when 'maxTotalHealthcheckTimeout' set
• 1575 - Clarify expected runtime, add execution time limit in ui
• 1569 - Remove extra scheduled entries from pending queue
• 1564 - Fix values for the container type field so they deserialize properly.
• 1553 - Release lock on bounce when bounced with no running instances
• 1554 - Refuse to add cleaning tasks to LB
• 1556 - Catch 404s when there is no task history in run now modal
• 1535 - Ensure path is set on read endpoint
• 1491 - Fix maxDeployIdSize and maxRequestIdSize validation

Documentation

• 1567 - Changes to SMTP documentation & connection
• 1548 - Fix markdown formatting
• 1528 - Rephrase flagging host

Thanks

• @darcatron
• @baconmania
• @PtrTeixeira
• @jaredstehler
• @peterlebrun
• @andrhamm

Singularity 0.15.1

Changes in 0.15.1

This is a bug fix release!

Check out the 0.15.1 milestone to see updates in detail.

Bug Fixes

• 1520 - Fix for request stuck in deleting state
• 1529 - Default cacheOffers to false and add docs
• 1530 - Fix for Optional request bodies

Documentation

• 1527 - Typo in documentation
• 1531 - Bump version numbers in documentation

Thanks

• @PtrTeixeira
• @darcatron
• @matteofigus

Singularity 0.15.0

Changes in 0.15.0

Check out the 0.15.0 milestone to see new features / bugfixes in detail.

New Features

• 1519 - Spread to all slaves

Adds a best effort attempt to deploy a task on all slaves. You can enable this by adding spreadAllSlavesEnabled: true to your Singularity yaml file. You can then set SPREAD_ALL_SLAVES for the placementStrategy on a SingularityRequest

• 1405 - Slave Usage Monitoring UI

In the Admin drop-down on the UI, there is now a Slave Usage monitoring page. This page contains visualizations about resource usages amongst all the slaves. The circular progress meters display the percent of total resources utilized on all slaves. The resources (cpu and memory) heat map displays percentage utilized per slave.

• 1456 - Bump to java 8/Jersey 2/Dropwizard 1

Singularity is now officially on java 8! As part of this upgrade we have also bumped some library versions. Most notably upgrading to Jersey 2 and Dropwizard 1

• 1253 - New Tailer

The Singularity log tailer got a makeover to improve performance and memory usage. Currently you can still toggle between old and new tailer versions, but this will be removed in future releases.

Improvements

• 1516 - Include deploy marker and oldest deploy step in state
• 1517 - More flexible match on filename from logrotate
• 1511 - Add ability to prefix all email subjects
• 1489 - Fix Task Search navigation and refresh
• 1510 - Add getTaskByRunIdForRequest to client
• 1483 - Switch not loaded to loading in first tailer screen
• 1503 - Don't show logs panel if task never running
• 1486 - Use web cache for api calls from ui
• 1478 - Add leader cache
• 1481 - Allow artifact list to be specified on deploys
• 1507 - Ability to provide HttpConfig for SingularityClient
• 1499 - Warn when removing a request with lb configs
• 1487 - Add a message to the request history when scaling
• 1474 - Allow expensive endpoints to be disabled for non-admins
• 1477 - Better zk cleanup for removed requests
• 1480 - Put a limit on number of slaves to decommission at once
• 1479 - Reduce timeout on requests to task sandboxes
• 1472 - Add spacing line at the bottom of logs
• 1476 - Better zk performance logging
• 1475 - Singularity scheduler lock logging
• 1465 - Pass job user as environment variable to task
• 1473 - More disabled actions for pollers
• 1466 - Add links between tailer versions
• 1450 - Allow new tabs from global search page
• 1452 - Add a task credit system
• 1400 - Add slave and task usage tracking inside singularity ZK
• 1447 - Set an optional max number of active tasks for ON_DEMAND requests
• 1442 - Display previous overridden cleanups
• 1451 - Additional disaster actions
• 1453 - Remove the async status update queue

Bug Fixes

• 1518 - Be sure to close Graphite properly
• 1505 - Don't include cleaning tasks in instance count
• 1513 - Write files in subdirectories to splat path
• 1504 - Don't redirect until done fetching active tasks
• 1502 - MD5 is case insensitive
• 1492 - Correctly redirect in ui when no instances are found
• 1498 - Forbid health checks longer than kill time
• 1501 - Allow copying from the JSON button dialog
• 1490 - Mark as not bouncing if paused before bounce completes
• 1470 - Fix tailer when reloading file
• 1469 - Add jita access for updating readWriteAccessGroup
• 1467 - Fix task direct link to logs
• 1428 - Prevent flapping slave from rejoining cluster
• 1443 - Ignore 404s for /priority/freeze endpoint

Documentation

• 1471 - Add in a note about running singularity on docker for mac

Thanks

• @ssalinas
• @PtrTeixeira
• @wsorenson
• @tpetr
• @felixgborrego
• @wolfd
• @Friss

Singularity 0.14.1

Changes in 0.14.1

This is a bug fix release.

Check out the 0.14.1 milestone to see new features / bugfixes in detail.

Bug Fixes

• 1458 - Remove expiring scale on deploy finish
• 1457 - Don't allow multiple bounces for the same request
• 1461 - Interchange argument order in rootComponent
• 1462 - extract in mesos uri defaults to true, executable defaults to false

Documentation

• 1459 - fix README formatting

Thanks

• @PtrTeixeira
• @joezirilli

Singularity 0.14.0

NOTE!! - This release has a known bug with the Singularity UI. Please use 0.14.1 instead

Changes in 0.14.0

Check out the 0.14.0 milestone to see new features / bugfixes in detail.

Important

The next release of Singularity (0.15.0) will include an upgrade to java 8.

Configuration Changes

#1391 includes a rework of some of the S3 settings in Singularity. If you use the SingularityExecutor or SingularityExecutorCleanup modules and use the S3 upload features, you will need an update to your configuration. The fields for specifying which files to upload have been moved out of the SingularityExecutor. An example below shows all fields that would move.

Old Configuration which gets removed from SingularityExecutor and SingularityExecutorCleanup yaml files)

executor:
  s3UploaderBucket: my-logs-bucket
  s3UploaderKeyPattern: "%requestId/%Y/%m/%taskId_%index-%s-%filename"
  s3UploaderAdditionalFiles:
    - access.log
  s3StorageClass: "STANDARD_IA"
  applyS3StorageClassAfterBytes: 75000

New Configuration (if not already present for use with S3 log searching)

ADD TO SingularityExecutorCleanup YAML

# in SingularityExecutorCleanup yaml configuration
executorCleanup:
  defaultS3Bucket: my-logs-bucket
  s3KeyFormat: "%requestId/%Y/%m/%taskId_%index-%s-%filename"
  s3StorageClass: "STANDARD_IA"
  applyS3StorageClassAfterBytes: 75000
  s3UploaderAdditionalFiles:
   - filename: access.log
     # The default directory in the executor was set to 'logs', now it must be manually specified
     # If not specified, the directory to search for log files will be the task app directory in the sandbox
     directory: logs

ADD TO SingularityService YAML

s3:
  s3Bucket: my-logs-bucket
  s3KeyFormat: "%requestId/%Y/%m/%taskId_%index-%s-%filename"
  s3StorageClass: "STANDARD_IA"
  applyS3StorageClassAfterBytes: 75000
  s3UploaderAdditionalFiles:
   - filename: access.log
     # The default directory in the executor was set to 'logs', now it must be manually specified
     # If not specified, the directory to search for log files will be the task app directory in the sandbox
     directory: logs

NOTE - To upgrade smoothly, it is strongly recommended to deploy SingularityService and the SingularityExecutorCleanup before deploying the SingularityExecutor

New Features

• 1306 - Smarter Healthchecks

Singularity healthchecks are now split into two phases. Previous settings will continue to function but are deprecated. Health check options are now specified in the healthcheck object on the SingularityDeploy See The new documentation for more on updates to healthchecks.

Improvements

• 1435 - Updated mysql index on startedAt
• 1449 - Improve status logging
• 1434 - Match mesos artifact defaults
• 1424 - Rename metric for immediate uploaders
• 1427 - Ability to specify params on request history in client
• 1429 - Standardize on toString/hash/equals formats
• 1402 - Introduce offer cache to allow better resource allocation
• 1399 - Add ability to upload files immediately to S3
• 1417 - Ability to specify cache on mesos artifact
• 1377 - Report currentActiveInstances on SingularityDeployProgress
• 1410 - Use an ldap cache
• 1414 - Add style to change display of overflow text in run now dialog
• 1415 - Replace existing global search with search from requests page
• 1411 - Logging clean up for mesos protos
• 1334 - Sanitize file data before parsing with Jackson
• 1386 - Add a DELETEING state for reqeusts being deleted
• 1388 - send graphite datapoints with optional tags
• 1391 - S3 Search Improvements
• 1401 - Ability to override docker workdir
• 1398 - Clarifications on S3ArtifactSignature
• 1376 - Update request for 'deploy to unpause' before saving pending deploy
• 1397 - Shade com.google.thirdparty
• 1375 - Allow setting S3 storage class at upload time
• 1373 - customizable --use-compress-program for tar
• 1392 - Optionally skip addition of extra s3 metadata
• 1382 - Add maxTasksPerOffer at request level
• 1367 - Additional settings for history purger
• 1360 - Bounce updates for placement and scaling
• 1369 - Additional threshold for deleting task history row data in sql

Bug Fixes

• 1437 - Defer loading log files till after page load
• 1445 - use implicit acks when not offloading status updates to another thread
• 1433 - Properly roll back from an overdue incremental deploy
• 1426 - Prevent building duplicate immediate uploaders
• 1412 - fix duplicate exit checker, add longer initial task wait
• 1423 - Update docker parameters on deploy form
• 1409 - Properly allow destroy task from ui
• 1407 - Forbid more characters from request/deploy IDs
• 1418 - Fixes for Immediate Uploaders
• 1403 - Import formModal on racks page
• 1371 - S3Downlaoder - Block on download only
• 1387 - No retry for scheduled tasks run on-demand through UI
• 1380 - Remove compressed log viewing endpoint
• 1381 - remove all compressed log viewing code
• 1372 - Update wrapper script for shell commands to read correct pid
• 1368 - Read group from task data if present

Documentation

• 1365 - Fix PATH in API annotations

Thanks

• @tpetr
• @wsorenson
• @darcatron
• @PtrTeixeira
• @jhaber
• @stevegutz

Logfetch 0.29.3

Logfetch 0.29.3 (caa8e82) fixes the following bugs:

• Be sure to keep start/end time metadata fro individual task api calls
• log a warning when tasks searched are limited by the max task_count (with instructions for args to search all remaining tasks)
• fix the date overlap logic so matching tasks for a search are narrowed down better

Logfetch 0.29.2

Logfetch 0.29.2 fixes the following bug:

#1393 - Fix possible timeout when listing s3 logs at request level with additional metadata. Makes use of the query param implemented in #1392

Logfetch 0.29.1

Logfetch 0.29.1 fixes the following bug:

• #1389 - Referenced before assignment error when searching s3 logs

Logfetch 0.29.0

Logfetch 0.29.0 contains the following updates:

• When available, use the startTime and endTime metadata for S3 log files to better filter more specific time ranges
• Do not show the progress bar when in verbose mode