v24.11.2

What's Changed

• [SCHEMATIC-136] Include schematic_api to resolve dependency issue by @thomasyu888 in #1547

Full Changelog: v24.11.2...v24.11.2

v24.11.2-rc

What's Changed

• [SCHEMATIC-136] Include schematic_api to resolve dependency issue by @thomasyu888 in #1547

Full Changelog: v24.11.1-rc...v24.11.2-rc

See https://github.com/Sage-Bionetworks/schematic/releases/tag/v24.11.1 for more changes in the v24.11 release series

v24.11.1

NOTE: there is a bug with this release in the CLI

 schematic --help
Traceback (most recent call last):
  File "/Users/tyu/anaconda3/envs/schematic-testing/bin/schematic", line 5, in <module>
    from schematic.__main__ import main
  File "/Users/tyu/anaconda3/envs/schematic-testing/lib/python3.9/site-packages/schematic/__init__.py", line 23, in <module>
    from schematic_api.api.security_controller import info_from_bearer_auth
ModuleNotFoundError: No module named 'schematic_api'

What's Changed

FEATURES

• [FDS-2386] Synapse entity tracking and code concurrency updates by @BryanFauble in #1505
• Attach additional telemetry data to OTEL traces by @BryanFauble in #1519
• feat: added tracing for cross manifest validation and file name validation by @linglp in #1509
• [SCHEMATIC-126] Update file annotation store process to require filename be present i… by @BryanFauble in #1535

BUG FIXES

• [FDS-2506] Fix type Validation by @andrewelamb in #1526
• [SCHEMATIC-1] BugFix: manifest submission when data previously annotated without manifest upload by @GiaJordan in #1538

TECH DEBT

• Update CODEOWNERS by @andrewelamb in #1525
• Update CODEOWNERS and pin down trivy version by @thomasyu888 in #1531
• Fds 2333 validate attribute unit tests by @andrewelamb in #1517
• [FDS-2347] Add unit tests to DataModelValidator class by @andrewelamb in #1523
• [FDS-2466] Update references to schema location in config.yml by @jaymedina in #1522
• Updating contribution doc to expect squash and merge by @BryanFauble in #1534
• [FDS-2491] Integration tests for Schematic API Test plan by @BryanFauble in #1512
• [FDS-2500] Add Integration Tests for: Manifest Validation by @jaymedina in #1516
• [FDS-2449] Lock sphinx version and update poetry.lock by @jaymedina in #1530
• Update for integration test failures by @BryanFauble in #1537
• [FDS-2525] Authenticated export of telemetry data by @BryanFauble in #1527
• [FDS-2238, FDS-2236] Automate CLI test plan by @andrewelamb in #1528
• [SCHEMATIC-157] Make some dependencies required to avoid schematic CLI commands from potentially erroring when doing a pip install by @thomasyu888 in #1540

Full Changelog: v24.10.2...v24.11.1

v24.11.1-rc

What's Changed

FEATURES

• [FDS-2386] Synapse entity tracking and code concurrency updates by @BryanFauble in #1505
• Attach additional telemetry data to OTEL traces by @BryanFauble in #1519
• feat: added tracing for cross manifest validation and file name validation by @linglp in #1509
• [SCHEMATIC-126] Update file annotation store process to require filename be present i… by @BryanFauble in #1535

BUG FIXES

• [FDS-2506] Fix type Validation by @andrewelamb in #1526
• [SCHEMATIC-1] BugFix: manifest submission when data previously annotated without manifest upload by @GiaJordan in #1538

TECH DEBT

• Update CODEOWNERS by @andrewelamb in #1525
• Update CODEOWNERS and pin down trivy version by @thomasyu888 in #1531
• Fds 2333 validate attribute unit tests by @andrewelamb in #1517
• [FDS-2347] Add unit tests to DataModelValidator class by @andrewelamb in #1523
• [FDS-2466] Update references to schema location in config.yml by @jaymedina in #1522
• Updating contribution doc to expect squash and merge by @BryanFauble in #1534
• [FDS-2491] Integration tests for Schematic API Test plan by @BryanFauble in #1512
• [FDS-2500] Add Integration Tests for: Manifest Validation by @jaymedina in #1516
• [FDS-2449] Lock sphinx version and update poetry.lock by @jaymedina in #1530
• Update for integration test failures by @BryanFauble in #1537
• [FDS-2525] Authenticated export of telemetry data by @BryanFauble in #1527
• [FDS-2238, FDS-2236] Automate CLI test plan by @andrewelamb in #1528
• [SCHEMATIC-157] Make some dependencies required to avoid schematic CLI commands from potentially erroring when doing a pip install by @thomasyu888 in #1540

Full Changelog: v24.10.2...v24.11.1-rc

v24.10.2

What's Changed

FEATURES

• feat: set annotations in an async way when submitting a manifest -- new changes added by @linglp in #1452
• Feature: filenameExists Validation Rule by @GiaJordan in #1456
• feat: added mermaid diagrams for some functions in ManifestGenerator by @linglp in #1463

BUG FIXES

• fix: updated schematic installation instructions by @linglp in #1462
• fix: check existing great expectation suite before adding a new one by @linglp in #1458
• [FDS-2294] Prevent including project name twice in walked path by @BryanFauble in #1474
• Bugfix: async tests skipped during regular testing runs by @GiaJordan in #1479
• fix: updated test after updating to synapse python client 4.4.0 and updated test after test resources get updated on synapse by @linglp in #1471
• fix: added unit test for get_node_validation_rules function by @linglp in #1483
• fix: update artifact to v4 by @linglp in #1500
• [FDS-2415] Fixes pytest CI Python Versions by @BWMac in #1502
• [FDS-2415] Fixes pdoc Workflow by @BWMac in #1503
• fix: call get_node_validation_rules with appropriate parameters when storing annotations by @GiaJordan in #1478
• Fixes failing pdoc deployment by @BWMac in #1504
• update how annotations are accessed by @GiaJordan in #1485
• Fix: remove caching poetry dependencies in CI workfows by @linglp in #1507

TECH DEBT

• feat/technical debt: added trivy code scan to schematic repo by @linglp in #1448
• Enable google credentials tests by @GiaJordan in #1459
• update synapse login functionality by @GiaJordan in #1465
• update python client to 4.4.0 by @andrewelamb in #1468
• [FDS-2127] Update URL validation to use requests.options to verify connectivity by @BryanFauble in #1472
• [FDS-2294] Run schematic_api tests by @BryanFauble in #1476
• add csv data model to unit tests by @lakikowolfe in #1464
• Revert "add csv data model to unit tests" by @andrewelamb in #1488
• Added unit tests for validate attribute class by @andrewelamb in #1486
• Update existing file paths in manifests at generation to conform to new convention by @GiaJordan in #1467
• [FDS-2373] Update jwt verification by @BryanFauble in #1493
• [FDS-2373] Remove the need to do version checking on Synapse creation by @BryanFauble in #1495
• Change how dataset to validate against is specified for filenameExists rule by @GiaJordan in #1491
• [FDS-305] Prevent shared snypase resources during integration tests by @BryanFauble in #1482
• [FDS-2383] Update to run integration tests on default runner with python 3.10 by @BryanFauble in #1498
• FDS-2403 Add Codeowners file by @andrewelamb in #1501
• [FDS-1725] Missing entityId handling testing by @BWMac in #1496
• [FDS-2502] Walk back up directory tree to get location in project by @BryanFauble in #1518
• [FDS-2497] Wrap google API execute calls with a 5 attempt retry by @BryanFauble in #1513
• [FDS-2218] Update the README by @jaymedina in #1510

---

• Schematic v24.10.1 by @lakikowolfe in #1514
• v24.10.2 by @lakikowolfe in #1520

New Contributors

• @jaymedina made their first contribution in #1510

Full Changelog: v24.7.2...v24.10.2

v24.7.2

What's Changed

Bug Fixes:

• fix: used environment variable to control tracing which allows tracing to be off during pytest run and normal API run by @linglp in #1455
• fix: install patch version of synapse python client by @linglp in #1457
• v24.7.2 release by @lakikowolfe in #1461

Full Changelog: v24.7.1...v24.7.2

v24.7.1

Features

• Automatically save the JSON Validation schema by @mialy-defelice in #1415
• feat: surfaced schematic exceptions in flask APIs by @linglp in #1438
• feat: added tracing to schematic code base for manifest generate and manifest submission by @linglp in #1430

Bug Fixes

• Bugfix: FDS-1686 test manifest being changed by @andrewelamb in #1412
• BugFix: Manifest version increment by @GiaJordan in #1435
• BugFix: Files no longer added to records based manifests by @GiaJordan in #1447

Technical Debt

• Develop FDS 243 mypy schema module by @andrewelamb in #1416
• added pydantic to toml file by @andrewelamb in #1419
• update pandas and numpy by @andrewelamb in #1421
• FDS-511 removed typing extensions lock by @andrewelamb in #1432
• technical debt: Remove service_acct_creds_synapse_id from schematic config by @linglp in #1436
• Add maintainers and version info to AWS Docker image by @GiaJordan in #1437
• technical debt: run black for schematic api and test and modify pre-commit by @linglp in #1442
• technical debt: added unit test for changeFileMetaData function by @linglp in #1444
• fix: added black compatibility to isort by @linglp in #1453
• technical debt: added isort to pre-commit hook in schematic repo by @linglp in #1441

Full Changelog: v24.5.1...v24.7.1

v24.5.1

Bug Fix

• BugFix: JSON Schema Validation Issues by @mialy-defelice in #1420
• technical debt: remove release.md from repo by @linglp in #1426
• Bug Fix: Fix failing test to conform to data model updates by @mialy-defelice in #1428

Full Changelog: v24.4.1...v24.5.1

v24.4.1

Features

• Allow non-entry if Required=False, while still enforcing validation rules if value is entered by @mialy-defelice in #1390
• Add Manifest download to the CLI by @mialy-defelice in #736
• Cross Manifest Validation: Allow first manifest to submit, if no target manifest to validate against. by @mialy-defelice in #1398
• Reorganize validation message level determination as a precursor to single rule disjunction by @GiaJordan in #1366
• Match None Manifest Validation Rule FDS-1766 by @mialy-defelice in #1387
• Add functionality to set requirements based per component by @mialy-defelice in #1401
• Updated python client to ^4.1.0 by @andrewelamb in #1403

Bug fixes

• Updated workflows to make sure uwsgi gets installed when publishing an image by @linglp in #1377
• Updated workflow to install dependencies and root project together by @linglp in #1380
• Schematic v24.2.1 AWS deployment fix by @linglp in #1378
• Schematic v24.2.1 AWS deployment fix - 2 by @linglp in #1382
• Fix wrong function name reference in function get_viz_component_attributes_explorer by @linglp in #1384
• Added installation of uWSGI in docker file by @linglp in #1381
• Avoid repeatedly calling generate_data_model_graph by @linglp in #1374
• Fix check_synapse_cache_size function; allow file size to be float by @linglp in #1389
• Added help text for file-annotations-upload parameter by @linglp in #1406
• Remove schematic init from CLI by @linglp in #1400
• Added project scope parameter in test_api.py by @linglp in #1409
• JSONLD dictionary parsing error by @mialy-defelice in #1407
• Convert nans in manifest to empty strings just prior to JSON Schema Validation by @mialy-defelice in #1402

Technical debt

• Mypy visualization module by @andrewelamb in #1373

• Mypy utils files by @andrewelamb in #1375

• Pylint schema utils by @andrewelamb in #1371

• Add pre commit hook for black by @GiaJordan in #1379

• Remove makrupsafe from toml file by @andrewelamb in #1383

• Remove temp data model references from test_api by @mialy-defelice in #1386

• Develop mypy schema utils by @andrewelamb in #1392

• Add synapse cache as a parameter to SynapseStorage by @andrewelamb in #1394

• Added file-annotations-upload parameter when submitting a manifest by @linglp in #1395

• Develop pylint schema module by @andrewelamb in #1393

• Turn pyopenssl package optional by @linglp in #1413

• Remove uneeded parameter from submit_metadata_manifest by @andrewelamb in #1410

• Schematic v24.4.1 by @lakikowolfe in #1417

New Contributors

• @lakikowolfe made their first contribution in #1417

Full Changelog: v24.2.1...v24.4.1

v24.2.1

What's Changed

Features:

• Retain formatting of Display name for annotations and table column names by @mialy-defelice in #1330
• Data Model Labels, display_label or class_label options by @mialy-defelice in #1356
• Component Based Rule Settings by @mialy-defelice in #1368

Bug Fixes

• feature-configuration-fixes by @andrewelamb in #1357
• Fixed an issue where empty rows would not be removed from manifests, and add test for load_df util function by @GiaJordan in #1346
• Increase schematic API time out by @linglp in #1358
• Deleted old version of readthedocs yaml file by @linglp in #1365

Technical Debt

• Updated poetry version in readthedocs.yml by @linglp in #1364
• Update pandas and schematic_db dependencies by @GiaJordan in #1347
• Run rule combinations tests at every push instead of only selectively by @GiaJordan in #1363
• Linted visualization module by @andrewelamb in #1359
• Move remained logic in routes.py related to manifest generator to library by @linglp in #1351
• Update parallel_map to parallel_applymap by @mialy-defelice in #1372
• Schematic v24.2.1 by @GiaJordan in #1376
• Make api packages optional by @andrewelamb in #1362
• Linted utils module by @andrewelamb in #1361
• Ran black on all modules by @andrewelamb in #1355
• Update synapse client minimum version to 3.2.0 by @GiaJordan in #1345

Full Changelog: v24.1.1...v24.2.1