Sprint 110: Release Report (sprint-109 - sprint-110)

Major accomplishments:

• Ingest releases addressing: Local ID lock, new state, reduced threads, notification title update
• Inventory release: Fix ephemeral Zookeeper lock
• UI release: Fuzzy search
• ETDv1 release: SQL updates to optimize CSV report generation, KeyError campus abbrv fix
• New UCI Nuxeo direct deposits
• Restart regular Nuxeo deposits

mrt-doc sprint-109..sprint-110

• 42efa89 2024-11-06 Update sprint-log.md
• 9da33b1 2024-10-23 Update README.md
• 7cf05a7 2024-10-22 Update next_steps.md

mrt-doc-private sprint-109..sprint-110

• 2b81e36 2024-10-23 Update dependency_scans.md
• 6664eed 2024-10-22 Create dep.newer.txt
• b0348d1 2024-10-22 Update dependency_scans.md
• a9e6c58 2024-10-22 Update dependency_scans.md

mrt-inventory sprint-109..sprint-110

• 8539641 2024-10-28 Merge pull request 35 from CDLUC3/dupfix
• bfe9cd7 2024-10-28 Improve freeing lock 2
• 54ae978 2024-10-25 Improve freeing lock
• 0846ebf 2024-10-24 Expand lock timeout to one hour
• 6b0a131 2024-10-24 Get manifest after lock test

mrt-ingest sprint-109..sprint-110

• 168f092 2024-11-01 emove SLF Simple provider. Conflicts with Logback SLF4J(W): Class path contains multiple SLF4J providers. SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@708c9f9e] SLF4J(W): Found provider [org.slf4j.simple.SimpleServiceProvider@6cf5cb11]
• 42fa0fe 2024-10-23 Merge pull request 108 from CDLUC3/localidLock
• 6ebb6f5 2024-10-23 Only release localID lock if set
• 1bd2bb3 2024-10-22 New commit to trigger rebuild
• 83801a2 2024-10-21 - LocalID locking - Notification error for multiple Local IDs - Release lock if Acquire fails - Email subject fix for failure - Support Pending to Failed state (Manifest processing error)

mrt-zk sprint-109..sprint-110

• d70de77 2024-11-01 rubocop and gem updates
• af62734 2024-11-01 rm legacy methods
• 1a382de 2024-10-25 Update data.md
• 7963055 2024-10-25 Update data.md
• 3918865 2024-10-22 Merge pull request 6 from CDLUC3/localidLock
• a5752f7 2024-10-21 - Support LocalID locks - Support Pending --> Failed state (manifest processing error)

mrt-dashboard sprint-109..sprint-110

• b22c306b 2024-11-01 Merge pull request 164 from CDLUC3/oct2024a
• e8b5ce51 2024-11-01 load defaults change
• 0dcca5f4 2024-10-25 additional owner delete
• 8d673c7e 2024-10-25 clean up localids
• 3f461084 2024-10-25 clean up recs in rspec
• 87df585f 2024-10-25 config reset
• 1ec776ba 2024-10-25 nocov
• 564fbe34 2024-10-25 improve error handling
• bf961bf9 2024-10-25 improve error handling
• 454deb8c 2024-10-23 prevent guest user fuzzy search
• ba982434 2024-10-22 gem updates
• ee6fbe3e 2024-10-17 add tests for file wildcard
• 2be03cfb 2024-10-17 fix coverage tests
• 5a2d2f2c 2024-10-17 filename fuzzy search

mrt-admin-lambda sprint-109..sprint-110

• b4daad9 2024-11-04 reg update
• ece9f2f 2024-11-04 Merge pull request 64 from CDLUC3/rm_legacy
• 1cc6ab5 2024-11-04 cognito include
• 2f097e1 2024-11-04 elim cognito lambda
• 301302b 2024-11-01 gem, zk, rubocop updates
• ca33fd0 2024-11-01 eliminate legacy
• 6b61f7a 2024-10-31 save ldapcollmap rpt
• 0d50811 2024-10-22 gem updates, retry storage state req

mrt-cron sprint-109..sprint-110

• 032a9f8 2024-11-04 Update reports.yml
• 437d619 2024-10-31 Update reports.yml

mrt-sceptre sprint-109..sprint-110

• ccaa3e1 2024-11-05 increase memory and timeout
• edd8461 2024-11-05 add comment
• 68e4426 2024-11-05 Update index.md
• 0546dc3 2024-11-04 fix timeout
• 55f668a 2024-10-24 Merge branch 'main' of github.com:CDLUC3/mrt-sceptre
• 05bdae3 2024-10-24 admin-sinatra app

merritt-docker sprint-109..sprint-110

• 90df09b 2024-11-04 obsolete image
• 249c128 2024-11-01 commits and readme
• 0b87116 2024-10-31 add inv_tasks to table def
• dfa577a 2024-10-22 commits
• 736ab57 2024-10-22 gem updates
• e878a1d 2024-10-17 update commits

uc3-etds sprint-109..sprint-110

mrt-repo-tagger sprint-109..sprint-110

• 9c78ba7 2024-10-23 Update config.yml

mrt-dashboard-config sprint-109..sprint-110

• 32ed1e2 2024-10-29 #2028
• ecc1c37 2024-10-29 #2047
• 9857d25 2024-10-28 Bulk update of new base URL - old https://s3.amazonaws.com/static.ucldc.cdlib.org/merritt -new https://pad-nuxeo.s3.us-west-2.amazonaws.com/nuxeo_merritt/atom_feeds
• d0e35db 2024-10-28 Update base URL - test of Nuxeo upgrade

Deployment 2024-11-04b - Merritt UI fuzzy Search

#2081

Deployment 2024-11-04a - Ingest - Remove conflicting logging provider AND Decrease consumer threads

#2094

Deployment 2024-10-28 - Inventory - Fix ephemeral Zookeeper lock

#2072

Deployment 2024-10-24 - Ingest - Local ID Lock, New State, bug fixes

#2082

Deployment 2024-10-21 - ETDs v1 - Make campus CSV report sql queries consistent

#2077

Sprint 109: Release Report (sprint-108 - sprint-109)

Major accomplishments:

• Queue Redesign release: Ingest, Inventory, mrt-zk
• Stage collection and admin object cleanup completed
• Stage automated cleanup implemented for removing regression test content

mrt-doc sprint-108..sprint-109

• 5863d96 2024-10-16 Update sprint-log.md
• 7a6e0ec 2024-10-16 Update sprint-log.md
• 8aa7191 2024-10-03 Update next_steps.md
• f76d7cc 2024-10-03 dup section
• bbc9d72 2024-10-03 manual deploy - all 5 services
• 304e821 2024-10-03 manual deployment notes
• ff4f4ab 2024-10-03 Update manual-deploy.md
• e909a5e 2024-10-03 Create manual-deploy.md
• edf8bcb 2024-10-01 Update next_steps.md

mrt-doc-private sprint-108..sprint-109

• 92f406f 2024-10-09 Update routine_librato_checks.md
• 0959f4d 2024-10-08 Update dependency_scans.md

mrt-inventory sprint-108..sprint-109

• 7a83864 2024-10-14 Merge pull request 34 from CDLUC3/zoofix
• d84c9b3 2024-10-11 Fix ephemeral zoo locks
• b03350e 2024-10-03 Merge pull request 33 from CDLUC3/zooInv
• c529665 2024-10-03 restore back README
• 6cbb271 2024-10-03 dummy change to force rebuild
• 7f967fc 2024-10-01 Merge branch 'zooInv' of https://github.com/CDLUC3/mrt-inventory into zooInv
• 3e3c906 2024-10-01 Fail Job on bad manifest url
• 2cd0c58 2024-09-23 semantic tag handling
• 54b0433 2024-09-13 tag determination
• b73e430 2024-09-13 changes won't upload - Merge branch 'zooInv' of https://github.com/CDLUC3/mrt-inventory into zooInv
• 5fd8e1e 2024-09-13 Refactor ark lock handling
• 3e226b2 2024-08-30 build tag calcluation
• 35ca6d4 2024-08-29 build.content.txt
• dfe1363 2024-08-28 Add pending status
• fe02ce7 2024-08-28 tag reportupdate
• 335f857 2024-08-28 Add tast db handling
• e88b95c 2024-08-27 pom var
• 63fa64d 2024-08-27 use ECR_REGISTRY for images
• 2489eec 2024-08-27 publish conf
• 4ab04ad 2024-08-26 multiline format
• ba64a41 2024-08-26 reset war name
• 11e5224 2024-08-26 war name
• e687c84 2024-08-19 reponame
• 15cf127 2024-08-16 cleanup artifacts
• e0a9f62 2024-08-16 buildspec report
• 0ca243c 2024-08-16 test resource
• 4082488 2024-08-12 fix build label
• 5f62633 2024-08-12 tagging rules
• e834d01 2024-08-12 tagging rules
• ba527fc 2024-08-12 rm create ecr
• 2042768 2024-08-06 buildspec normalize
• 47a91e0 2024-07-31 Merge pull request 32 from CDLUC3/codebuild2
• 1d0908d 2024-07-31 zooInv
• ebdfbc4 2024-07-30 version refs
• 4ed3908 2024-07-29 3.0-SNAPSHOT
• f4f0943 2024-07-26 build
• a6ee6ec 2024-07-25 zk snapshot
• b9a2620 2024-07-25 buildx
• a40d74a 2024-07-25 multiplatform
• afd61cc 2024-07-24 publish invconf
• 15ef9a9 2024-07-24 eliminate inventoryconf
• 97c60a9 2024-07-24 build content fix
• 6ef13cb 2024-07-24 codebuild2
• 8c2ee6e 2024-07-24 tidy buildspec and jdpa
• 1ea1381 2024-07-24 INV it tests
• 1fbe3b3 2024-07-24 pom
• 873f210 2024-07-24 gitignore
• 6fb50a7 2024-07-22 refine build
• 9a30463 2024-07-22 inv buildspec
• e0f886a 2024-07-18 codebuild
• 1fa68ac 2024-07-23 Remove references to cdl-zk-queue
• 1001c95 2024-07-22 refine build
• 603df21 2024-07-22 inv buildspec
• a5a1dcd 2024-07-18 codebuild
• d11782c 2024-06-28 SaveObject zooKeeper close
• e4ca0a1 2024-06-27 Merge pull request 31 from CDLUC3/zooInv2
• 952bb1c 2024-06-27 one more cleanup
• 4f31c70 2024-06-27 Remove System.out.println s
• 74534f9 2024-06-27 Restore IT test for adding object through zk zookeeper
• ba33c75 2024-06-26 IT test with ZooKeeper directly
• 5838549 2024-06-26 better track state
• 67dd329 2024-06-24 initialize zookeeper nodes
• c7d493f 2024-06-21 inv new zoo
• 6aeeebf 2024-06-21 inv new zoo
• 4f8982c 2024-06-21 inv new zoo

mrt-ingest sprint-108..sprint-109

• d87dff8 2024-10-16 Merge pull request 107 from CDLUC3/jobCleanup
• fe467af 2024-10-16 Turn off hourly Job Cleanup
• 9819ff3 2024-10-15 Merge pull request 106 from CDLUC3/localidNotify
• 100e594 2024-10-14 - Refresh Job config/ids after processing. - Ensure notification gets latest LocalID data
• b2db03b 2024-10-07 Fix serious Update/Submit bug!
• 717c3fc 2024-10-03 Merge pull request 105 from CDLUC3/queue_LIB
• 11d4309 2024-10-02 #2046
• 4532d02 2024-10-02 Fix logic error for ZFS usage
• 73c9d5d 2024-10-02 Support ZFS usage check in Provision Handler
• 65e31e6 2024-10-01 Trap more Zookeeper exceptions for long running sessions
• 9a9e079 2024-09-29 - Force a bad Storage manifest for Inventory on demand - Add submissionDate to Job Configuration
• e78cc12 2024-09-26 Remove deleted jobs
• 3320f66 2024-09-25 Force failure if file exists. Useful for testing
• ecbc764 2024-09-25 Be more clear on Cleanup process scheduling
• a97aa20 2024-09-24 Trap Connection loss and Session Expiration and retry
• 1be0dbb 2024-09-23 semantic tag handling
• deaf215 2024-09-23 Ensure that we unlock job when switching to Held state
• abe5e3c 2024-09-18 Reconnect after session expiration for all Consumer Daemons
• 1cc73e3 2024-09-13 Cosmetic change. Trigger a rebuild.
• 1ec5d18 2024-09-13 update tag logic
• bbd744c 2024-09-13 TAG logic
• c4d596b 2024-09-12 Fix logic error with queue status in State command
• d26b25f 2024-09-09 Specify FAIL on Batch and Job errors
• 6f7d27a 2024-09-05 Control Zookeeper logging with logback
• 29e1ed0 2024-08-30 Set default LOGLEVEL to warn (from INFO)
• 0f29c76 2024-08-30 build tag calcluation
• 44ff474 2024-08-29 build.content.txt
• 1813c7a 2024-08-28 Fix serviceStartTime within state. Add more IT assertions w/ primaryID
• 4bb3b88 2024-08-28 Merge branch 'queue_LIB' of github.com:cdluc3/mrt-ingest into queue_LIB
• fe4ca06 2024-08-28 tag report update
• 94c3527 2024-08-27 Add LocalID Integration testing w/ ERC/Form/Manifest entry points
• 0452f8f 2024-08-27 pom var
• 7028afd 2024-08-27 use ECR_REGISTRY for images
• fbe982e 2024-08-22 pull through cache
• f1bf99c 2024-08-22 registry vars
• f338489 2024-08-22 enable tests
• 327c93f 2024-08-22 img refs
• a524017 2024-08-21 fakesmtp image
• 3a3d526 2024-08-19 reponame
• 08b1368 2024-08-16 buildspec
• e1daf70 2024-08-16 clean unlisted
• 51aeeaf 2024-08-16 format buildspec
• e809a90 2024-08-16 buildspec
• 3321dad 2024-08-16 update integ tests and buildspec
• 69cc8e9 2024-08-16 test driver fix
• 2edb0a4 2024-08-16 Add Initialization Consumer Daemon
• 2943c53 2024-08-12 fix 6
• c20e37d 2024-08-12 fix 5
• 72530a4 2024-08-12 fix 4
• 20d45e3 2024-08-12 fix 3
• 9ddba6c 2024-08-12 fix 2
• 0ce10d4 2024-08-12 fix
• d8d5957 2024-08-12 commit date for build info
• 3858e23 2024-08-12 refine tagging
• 29017cb 2024-08-12 rm ecr create
• 156a0e1 2024-08-06 buildspec normalize
• 4166f4c 2024-07-31 Merge pull request 104 from CDLUC3/codebuild
• a4071fd 2024-07-31 queue_LIB
• 697927e 2024-07-30 version refs
• afe12b5 2024-07-29 3.0-SNAPSHOT
• 952d001 2024-07-26 build
• 819fecf 2024-07-25 zk snapshot
• 542115b 2024-07-25 buildx
• 391c0a4 2024-07-25 multiplatform
• 44f2fd2 2024-07-24 docker profiles
• 982f525 2024-07-23 pom consistency
• 2c5eb0d 2024-07-22 it
• fc78329 2024-07-22 docker
• 73558bf 2024-07-19 dockerfile fix
• 4ad4e3b 2024-07-19 dockerfile
• 81762eb 2024-07-19 docker fix
• 928e00f 2024-07-19 ingest war
• f8cf4f1 2024-07-19 skip tests
• 29a7b1f 2024-07-19 refine test driver
• 9de3f3c 2024-07-18 codebuild fakesmtp
• c27f610 2024-07-18 use ecr
• 3103c9e 2024-07-18 use own copy of fakesmtp
• 9315afe 2024-07-17 settings.xml
• ac1e821 2024-07-17 ingest buildspec
• 7bfc3d8 2024-07-08 Ensure that we do not exceed Thread Pool size when acquiring data
• bbcc58c 2024-07-03 Remove all reference and usage of old ZK library
• 7818153 2024-07-01 Remove all references of DistributedQueue from old ZK lib
• a8dde51 2024-06-27 Cosmetic
• 1939c58 2024-06-27 Fix session exception
• e4a05c9 2024-06-27 Was too aggressive with ZK closure
• 9b66944 2024-06-27 - Close all ZK client connections upon exit - Fix Manifest URL for Inventory
• ddb97e0 2024-06-26 Support collection level freeze with new ZK lib
• 81d4f3b 2024-06-25 Ingest Queue Hold support with new ZK lib
• f7c00d7 2024-06-21 Support Locks with new ZK lib
• 62afc38 2024-06-20 Force notification attachment format of queueing to JSON
• 00ddd99 2024-06-20 Clean up debug statements
• 3d1bcd9 2024-06-18 Cosmetic
• bb45b23 2024-06-14 More fault tolerant to exceptions
• 38eeb5d 2024-06-12 Fail Batch if Job posting fails Creat job with priority only for Estimation state
• 84c594d 2024-06-12 Remove concept of Jobs States (plural)
• 1bcf91b 2024-06-12 Remove JobsState and /status API
• be984f3 2024-06-12 - Populate Inventory ZK field - More fault tolerant for ZK setStatus
• 5c17c9f 2024-06-04 use mrt-zk for IT test validation
• cbd6aa2 2024-06-03 Look for Completed state
• f51269b 2024-06-03 old zk lib dependency
• 2b6584b 2024-06-03 test cleanup
• 107a8b9 2024-06-03 New handlers
• eed98cb 2024-06-03 it test changes
• 75df29c 2024-05-31 Use new ZK method to initialize root ZK paths.
• 2234ca0 2024-05-31 "submit-object" and "update-object" endpoints deprecated
• 289805a 2024-05-29 Support state command
• b1188f4 2024-05-28 mrt-json lib brings in version of JSON lib not supporting JSONObject.isEmpty()
• 16862cb 2024-05-24 Cosmetic
• 64991c7 2024-05-23 Store metadata and id info in CONFIGURATION JSONObject
• df09301 2024-05-21 Populate metadata in ZK lib
• 432fa4a 2024-05-21 Populate and utilize ids in ZK lib
• 1371715 2024-05-20 - Add Job failure status in Notification attachments and email message
• d0f4106 2024-05-19 Additional ZK work
• 1ce12fc 2024-04-20 Add Estimate and Provision daemon/process
• 5846676 2024-04-11 Process Batch queue
• e7852c7 2024-04-08 Post Batch data

mrt-zk sprint-108..sprint-109

• df72336 2024-10-08 Update states.md
• 83e808d 2024-10-08 batch job count
• 80fbed2 2024-10-08 list_batches
• f359186 2024-09-27 Update data.md
• d93d5dd 2024-09-27 dont set last successful to failed
• 52a8795 2024-09-27 dont set last successful to failed
• ebc36d2 ...

Deployment 2024-10-07 - Ingest, Inventory, mrt-zk - Queue Redesign Release

#2048

Sprint 108: Release Report (sprint-107 - sprint-108)

Major accomplishments:

• Ingest workspace introduced
• In-depth testing of queue redesign on Stage
• Additional progress on changeToken object update implementation

mrt-doc sprint-107..sprint-108

• ffe955b 2024-09-25 Update sprint-log.md
• 752a505 2024-09-25 Update README.md
• 5fe6888 2024-09-25 Create README.md
• b3461d9 2024-09-25 Update README.md
• 77ccb47 2024-09-18 Update README.md
• 64bc182 2024-09-18 Update README.md

mrt-audit sprint-107..sprint-108

• 5592eaa 2024-09-23 tag resolution regex
• a5c4f5f 2024-09-23 Force update for new semantic
• 680f3b5 2024-09-13 tag determination
• b2363ce 2024-09-10 empty commit to I can creat a new tag

mrt-replic sprint-107..sprint-108

• 46f46b8 2024-09-23 semantic tag handling
• 4fb497d 2024-09-13 tag determination

mrt-store sprint-107..sprint-108

• 9998f4c 2024-09-23 semantic tag handling
• 2f7b868 2024-09-13 tag determination

mrt-cloud sprint-107..sprint-108

• d5bcc0e 2024-09-23 semantic tag handling
• b2be015 2024-09-13 tag determination

mrt-core2 sprint-107..sprint-108

• 3617b66 2024-09-23 semantic tag handling
• 6441977 2024-09-13 tag determination

mrt-zk sprint-107..sprint-108

• ede861e 2024-09-25 Catch all Exceptions when cleaning the queue
• e7c837c 2024-09-25 Allow completion of Batch cleanup, even if bad node detected
• 1444828 2024-09-23 semantic tag handling
• 5f1afc7 2024-09-23 Update test_plan.md
• d992001 2024-09-13 Merge branch 'main' of github.com:cdluc3/mrt-zk
• b10a012 2024-09-13 tag determination
• eea3f4f 2024-09-13 Change language for not being able to delete Job/Batch

mrt-dashboard sprint-107..sprint-108

• 4c982715 2024-09-23 semantic tag handling
• 5b866cea 2024-09-13 tag determination

mrt-admin-lambda sprint-107..sprint-108

• 335df0c 2024-09-25 manage storage nodes
• 6c2e0f1 2024-09-25 secondary nodes report
• ef6c4cf 2024-09-25 fix secondary node report
• 63a7421 2024-09-13 tag determination

mrt-sceptre sprint-107..sprint-108

• b7a769c 2024-09-24 list buckets
• a12d2b8 2024-09-20 annotate sceptre templates
• 74eabda 2024-09-20 hardcode some MD content
• f13adec 2024-09-20 hardcode some MD content
• 38b79c1 2024-09-20 hardcode some MD content
• bcc81d0 2024-09-20 cors test
• 0d538c4 2024-09-20 tidy basic auth example, add hierarchy notes
• 28ca40d 2024-09-20 screen shot tidy
• 2916c3b 2024-09-20 screen shot tidy
• 9242959 2024-09-20 demo screenshots
• 414507e 2024-09-20 diagram updates
• bdc5d49 2024-09-19 Merge branch 'main' of github.com:CDLUC3/mrt-sceptre
• 775c2ee 2024-09-19 style box
• 9244b29 2024-09-19 Update README.md
• e544289 2024-09-19 documentation
• 11bd109 2024-09-19 tidy s3 module
• 06fe225 2024-09-19 tidy lambda module
• 09c1554 2024-09-19 elim old lambda config
• 3e522ac 2024-09-19 refine default config
• 45ea1c8 2024-09-19 demo project, better lambda template
• 9cde12c 2024-09-13 add TODO
• a077c0d 2024-09-13 del initial wkspace
• a48fc1c 2024-09-13 save notes
• 89bf5f0 2024-09-13 separate ec2 and fsx
• 40afadb 2024-09-11 remount fsx
• 23c7963 2024-09-11 refine user data
• 03a4b05 2024-09-11 ec2 and fsx security
• 8ba8edf 2024-09-10 fsx - work in prog
• 6e79a1b 2024-09-10 ec2 setup

merritt-docker sprint-107..sprint-108

• d944df7 2024-09-23 commits
• a0de6be 2024-09-23 semantic tag handling
• 8ea8c8d 2024-09-13 update commits
• 321a07c 2024-09-13 tag determination

mrt-repo-tagger sprint-107..sprint-108

• b787261 2024-09-25 Update config.yml
• 967cc04 2024-09-18 Update config.yml

Deployment 2024-08-27 - UI gem updates

#2014