LUC070-156 multithread deletes - implemented with integration and uni…#862
Draft
LUC070-156 multithread deletes - implemented with integration and uni…#862
Conversation
…858) * #LUC070-162 Fixed the bug where multiple reviews breaks the /vaults page 1) Updated ReviewsController so that /vaultreviews/depositreviews/ expects / uses a depositReviewId param rather than depositId 2) Updated DepositsReviewService so that getDepositReview now gets the depositReview by id rather than deposit_id 3) Updated VaultsController so that it passes the depositReviewId rather than deposit_id when getting the deposit review in getVault method 4) Updated RestService so that getDepostReview expected a depositReviewId rather than a depositId * #LUC070-162 Fixed the new bug where the fix for multiple reviews now breaks the /admin/vaults/vaultID/review page I think at some point we merged two lines of code into one, the fix for the previous issue changed the value of a param which then broke something else. I've put both params back and used them in the correct place. I suspect I'll have to fix / update some tests before this is complete but it all looks fine when deployed locally. I've also changed the order of the deposits in review output to be based on creation time. 1) Updated the order of the reviews list in VaultsController 2) Added both params (depositId and depositReviewId) to AdminReviewsController and updated the order of the reviews list in AdminReviewsController * #LUC070-162 Fixed tests that were failing after the fixes * #LUC070-180 Fix for the post 4.1.6 release review issue (#849) I noticed that the code was using the deposit id instead of DepositReviewId after changing that all the tests still pass. Things also look good when I did a test review on my local deploy. We'll need to test more on Demo. * Remove jacoco from build job as it appears to be failing after recent Jenkins upgrade. * Re add jacoco from build job as it didn't change anything (in fact I've realised it was working and providing a coverage report) * Remove jacoco from build job after uninstalling from Jenkins * More changes to try and debug the Jenkins demo build failure * More changes to try and debug the Jenkins demo build failure * Add the jacoco post run stuff back now we have discovered it was one of the docker plugins causing the issues. * DS-LUC070-145-20250318 Upgrade SB dependencies along with assorted other mvn dependencies (#848) * Spring Boot 3.3.2 and other library version upgrades updated maven wrapper too * #LUC070-145 Upgrade spring boot and other dependencies 1) Upgraded pom.xml 2) Upgraded various impacted tests * #LUC070-145 Upgrade spring boot and other dependencies 1) Upgraded pom.xml * #LUC070-145 Upgrade spring boot and other dependencies 1) Upgraded pom.xml * #LUC070-145 Upgrade spring boot and other dependencies 1) Upgraded pom.xml 2) Upgraded common/pom.xml Removed unused dep and upgraded tika * #LUC070-145 Upgrade spring boot and other dependencies 1) Upgraded pom.xml 2) Upgraded datavault-broker/pom.xml Removed unused dep and upgraded tika --------- Co-authored-by: David Hay <david.j.hay@gmail.com> * LUC070-209 - Added a local Accessibility page. (#852) Co-authored-by: John Pinto <jpinto@newington.edina.ac.uk> * Fix failing tests * Fix for Accessibility Statement footer links. They are now clickable. (#853) Co-authored-by: John Pinto <jpinto@newington.edina.ac.uk> * Luc070 209 Added local accessibility page (#854) * Fix for Accessibility Statement footer links. They are now clickable. * Fix for non-clickable links for footer on all pages except the "Edit Pending Vault" and "Create New Vault" pages. --------- Co-authored-by: John Pinto <jpinto@newington.edina.ac.uk> * LUC070-20 - Fix for making footer links clickable for Create Vault (#856) Pages. Co-authored-by: John Pinto <jpinto@newington.edina.ac.uk> --------- Co-authored-by: David Hay <david.j.hay@gmail.com> Co-authored-by: John Pinto <john.pinto@ed.ac.uk> Co-authored-by: John Pinto <jpinto@newington.edina.ac.uk>
* #LUC070-162 Fixed the bug where multiple reviews breaks the /vaults page 1) Updated ReviewsController so that /vaultreviews/depositreviews/ expects / uses a depositReviewId param rather than depositId 2) Updated DepositsReviewService so that getDepositReview now gets the depositReview by id rather than deposit_id 3) Updated VaultsController so that it passes the depositReviewId rather than deposit_id when getting the deposit review in getVault method 4) Updated RestService so that getDepostReview expected a depositReviewId rather than a depositId * #LUC070-162 Fixed the new bug where the fix for multiple reviews now breaks the /admin/vaults/vaultID/review page I think at some point we merged two lines of code into one, the fix for the previous issue changed the value of a param which then broke something else. I've put both params back and used them in the correct place. I suspect I'll have to fix / update some tests before this is complete but it all looks fine when deployed locally. I've also changed the order of the deposits in review output to be based on creation time. 1) Updated the order of the reviews list in VaultsController 2) Added both params (depositId and depositReviewId) to AdminReviewsController and updated the order of the reviews list in AdminReviewsController * #LUC070-162 Fixed tests that were failing after the fixes * #LUC070-180 Fix for the post 4.1.6 release review issue (#849) I noticed that the code was using the deposit id instead of DepositReviewId after changing that all the tests still pass. Things also look good when I did a test review on my local deploy. We'll need to test more on Demo. * Remove jacoco from build job as it appears to be failing after recent Jenkins upgrade. * Re add jacoco from build job as it didn't change anything (in fact I've realised it was working and providing a coverage report) * Remove jacoco from build job after uninstalling from Jenkins * More changes to try and debug the Jenkins demo build failure * More changes to try and debug the Jenkins demo build failure * Add the jacoco post run stuff back now we have discovered it was one of the docker plugins causing the issues. * DS-LUC070-145-20250318 Upgrade SB dependencies along with assorted other mvn dependencies (#848) * Spring Boot 3.3.2 and other library version upgrades updated maven wrapper too * #LUC070-145 Upgrade spring boot and other dependencies 1) Upgraded pom.xml 2) Upgraded various impacted tests * #LUC070-145 Upgrade spring boot and other dependencies 1) Upgraded pom.xml * #LUC070-145 Upgrade spring boot and other dependencies 1) Upgraded pom.xml * #LUC070-145 Upgrade spring boot and other dependencies 1) Upgraded pom.xml 2) Upgraded common/pom.xml Removed unused dep and upgraded tika * #LUC070-145 Upgrade spring boot and other dependencies 1) Upgraded pom.xml 2) Upgraded datavault-broker/pom.xml Removed unused dep and upgraded tika --------- Co-authored-by: David Hay <david.j.hay@gmail.com> * LUC070-209 - Added a local Accessibility page. (#852) Co-authored-by: John Pinto <jpinto@newington.edina.ac.uk> * Fix failing tests * Fix for Accessibility Statement footer links. They are now clickable. (#853) Co-authored-by: John Pinto <jpinto@newington.edina.ac.uk> * Luc070 209 Added local accessibility page (#854) * Fix for Accessibility Statement footer links. They are now clickable. * Fix for non-clickable links for footer on all pages except the "Edit Pending Vault" and "Create New Vault" pages. --------- Co-authored-by: John Pinto <jpinto@newington.edina.ac.uk> * LUC070-20 - Fix for making footer links clickable for Create Vault (#856) Pages. Co-authored-by: John Pinto <jpinto@newington.edina.ac.uk> * #LUC070-221 Fix the copy ssh key modal by making it open a tab instead of using an iframe (#857) Fix the copy ssh key modal by making it open a tab instead of using an iframe (#857) * #LUC070-228 Further changes re the accessibility page (#859) * #LUC070-228 Add lang="en" to the templates that don't have it. * #LUC070-228 Further changes re the accessibility statement --------- Co-authored-by: David Hay <david.j.hay@gmail.com> Co-authored-by: John Pinto <john.pinto@ed.ac.uk> Co-authored-by: John Pinto <jpinto@newington.edina.ac.uk>
…t tests. DeleteChunk events send per (chunk and location) deleted.
…eceived into Broker and saved to database
…ts' property in the broker can control whether the Workers send 'DeletedChunk' events. The default value is false. The value of that broker property gets send to worker as the 'workersSendDeletedChunkEvents' property in the Delete Task properties.
…vents in Broker from Workers
…th Archive in database
… into AdminDepositService
Collaborator
Author
|
Just pushed a fix - moved duplicated Delete Deposit code into AdminDepositService |
# Conflicts: # datavault-webapp/src/main/webapp/WEB-INF/templates/filestores/index.html # datavault-webapp/src/main/webapp/WEB-INF/templates/layout/footer.html
…itService was not being created.
* #LUC070-236 Post testing fixes 1) Tweaked overrideStyles.css so that the first header line is shown in Italics 2) Tweaked welcomelayout.html so there is a line of space between both header layout sections 3) Fixed the welcome page vault review date column being a different colour * #LUC070-236 Post testing fixes Fixed the resize issue with the header navbar and the footer.
* DS-LUC070-236-20260225 Post testing fixes * #LUC070-236 Post testing fixes 1) Tweaked overrideStyles.css so that the first header line is shown in Italics 2) Tweaked welcomelayout.html so there is a line of space between both header layout sections 3) Fixed the welcome page vault review date column being a different colour * #LUC070-236 Post testing fixes Fixed the resize issue with the header navbar and the footer. * #LUC070-241 Fix the headings and spacing between sections on the info fieldset.html (#865) Fix the headings and spacing between sections on the info fieldset.html (#865)
Collaborator
Author
|
Just merged in dv5-main branch tagged with 5.0.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding code to allow for Delete Task in worker to delete each archive chunk in parallel.
TODO
DONE
Multi-Thread Deletes Implementation
Supported Scenarios:
when a chunk or non-chunk archive file is deleted - a DeletedChunk event will be sent to the Broker - it has the following:
written deposit-then-delete integration tests:
written deposit-unit-tests to help simulate and test errors at random chunks. These all use test-specific ArchiveStore test device to control when error occurs