A feed of development activity from across Sia Foundation GitHub repositories.
This PR is a testing playground for a potential directories table. The goal is to find out whether a table like that can provide the index needed to significantly speed up fetching directory info on larger nodes with 500k+ objects.
- Create dirs when objects are created
- Delete empty dirs when objects are deleted
- Create/Delete dirs when objects are renamed
- SQLite migration code
- MySQL migration code
- Test performance before/after
UPDATE: I've experimented with multiple approaches so far. Directory table with full paths, with names only, recursive queries, non recursive ones, functional indices, generated columns with indices, partitioning, temporary tables and nested sets. The big problem seems to be our use-case of having 2 tables which don't just require a select but a group by on the sub-directories of the directory we want to query. According to some posts I have found about nested sets they were mostly used before MySQL supported recursive queries since recursive queries should perform the same for fetching immediate children and a lot faster for deep traversal. I confirmed that fetching the directories from the tree is quite fast at 10-20ms for a query that takes about 2 seconds in total using the recursive approach. However that part is fast for pretty much all approaches I have tested. The slow part is grouping the objects by parent dir which is pretty slow for every approach.
So I ended up choosing the fastest of all approaches I've tested which also happens to be the nicest query. A directory table that only contains the name of each directory instead of full path to avoid too much overhead. It takes about 800ms on my node with 500k objects and 150k directories in the higher level dirs compared to 2s for the fully recursive approach.
It is pretty straight forward and consists of 2 parts which are combined via UNION:
- We search for the directory that we want to query. Then we fetch all objects in that directory with a fast inner join.
- We search for all the child dirs with a quick inner join. Then we join all objects with these dirs using their paths before (and this is the slow part) grouping them by directory.
This PR adds errInvalidHandshake to the list of prune alert exception cases.
{ "id": "h:010f0c2e5180fce43e054aab2d3479bcd860f5da960c1277ebfb97999e858ec5", "severity": "warning", "message": "Contract pruning failed", "data": { "contractID": "fcid:5f69293916b6a5ddecef0d60347b3ef198ca623d8453d14e949becb2b15f884d", "error": "failed to prune contract fcid:5f69293916b6a5ddecef0d60347b3ef198ca623d8453d14e949becb2b15f884d; NewRenterTransport: couldn't read host's handshake: EOF\n", "hostKey": "ed25519:0220dabd125931d000c48ada38fb39082e0c7e241a1a469f2527a0846f692d32", "hostVersion": "1.6.0", "origin": "autopilot.autopilot" }, "timestamp": "2024-05-15T11:09:34.766139268Z" }
This PR expands test coverage in merkle to 95.1% and overall consensus coverage to 76.2% Now that we have the coverage for DecodeFrom and EncodeTo I'm planning to add a roundtrip later, assuming you find such tests to be valuable.
Adds support for a yaml configuration file similar to hostd and renterd.
directory: /etc/walletd autoOpenWebUI: true http: address: :9980 password: sia is cool consensus: network: mainnet gatewayAddress: :9981 bootstrap: false enableUPnP: false index: mode: partial # full, partial, none (full index will index the entire blockchain, partial will only index addresses that are registered in the wallet, none will treat the database as read-only and not index any new data) batchSize: 64 # max number of blocks to index at a time (increasing this will increase scan speed, but also increase memory and cpu usage) log: level: info # global log level stdout: enabled: true # enable logging to stdout level: debug # override the global log level for stdout enableANSI: false format: human # human or JSON file: enabled: true # enable logging to a file level: debug # override the global log level for the file path: /var/log/walletd.log format: json # human or JSON
Testing the migration on my node I found that we have to tweak the upload penalty in some cases.
This PR expands test coverage in policy.go to 86.5% and overall types coverage to 59.7%
Changes the read and write RPC to send internal errors and alert+log the actual error.
This PR adds a SQL database wrapper similar to the one in hostd which will eventually allow us to get rid of gorm.
The Backend and BackendMetrics interfaces are the interfaces that we will extend over time until we don't use gorm anymore. For the start the interfaces are only used to close the databases as well as get the version for printing.
One noteworthy change in functionality is that we now also use an inverted approach to retrying transactions. We no longer have a whitelist of errors for which we skip the retry but we always skip the retry unless a db specific error is encountered indicating the database is locked.
Fixes a small race introduced when the handling of tpool utxos changed
Bumps go.sia.tech/web/renterd from 0.52.0 to 0.52.1.
Release notes
Sourced from go.sia.tech/web/renterd's releases.
renterd@0.52.1
Patch Changes
- c89d2e55: Fixed an issue where the recommendations bar was overlapping notifications.
- c89d2e55: Fixed an issue where clearing some configuration fields would crash the app due to how recommendations were being calculated.
- c89d2e55: The configuration recommendations now instruct the user to first fill all fields. Closes SiaFoundation/renterd#1214
Commits
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps github.com/cloudflare/cloudflare-go from 0.94.0 to 0.95.0.
Release notes
Sourced from github.com/cloudflare/cloudflare-go's releases.
v0.95.0
ENHANCEMENTS:
- access_application: add support for
policiesarray (#1956)- access_application: add support for
scim_config(#1921)- access_policy: add support for reusable policies (#1956)
- dlp: add support for zt risk behavior configuration (#1887)
BUG FIXES:
- access_application: fix scim configuration authentication json marshalling (#1959)
DEPENDENCIES:
- deps: bumps dependabot/fetch-metadata from 2.0.0 to 2.1.0 (#1839)
- deps: bumps github.com/urfave/cli/v2 from 2.27.1 to 2.27.2 (#1861)
- deps: bumps golang.org/x/net from 0.24.0 to 0.25.0 (#1974)
- deps: bumps golangci/golangci-lint-action from 4 to 5 (#1845)
- deps: bumps golangci/golangci-lint-action from 5 to 6 (#1975)
Changelog
Sourced from github.com/cloudflare/cloudflare-go's changelog.
0.95.0 (May 8th, 2024)
ENHANCEMENTS:
- access_application: add support for
policiesarray (#1956)- access_application: add support for
scim_config(#1921)- access_policy: add support for reusable policies (#1956)
- dlp: add support for zt risk behavior configuration (#1887)
BUG FIXES:
- access_application: fix scim configuration authentication json marshalling (#1959)
DEPENDENCIES:
- deps: bumps dependabot/fetch-metadata from 2.0.0 to 2.1.0 (#1839)
- deps: bumps github.com/urfave/cli/v2 from 2.27.1 to 2.27.2 (#1861)
- deps: bumps golang.org/x/net from 0.24.0 to 0.25.0 (#1974)
- deps: bumps golangci/golangci-lint-action from 4 to 5 (#1845)
- deps: bumps golangci/golangci-lint-action from 5 to 6 (#1975)
Commits
4788db7Update CHANGELOG.md3f615f5update CHANGELOGd4a7f8dMerge pull request #1975 from cloudflare/dependabot/github_actions/golangci/g...a1fda3aMerge branch 'master' into dependabot/github_actions/golangci/golangci-lint-a...0026065Merge pull request #1974 from cloudflare/dependabot/go_modules/golang.org/x/n...04c2cebadd CHANGELOG for #1975e3ab1b2build(deps): bump golangci/golangci-lint-action from 5 to 6be510c4add CHANGELOG for #1974e35c25dbuild(deps): bump golang.org/x/net from 0.24.0 to 0.25.0a8e30d4Merge pull request #1959 from khiller-cf/master- Additional commits viewable in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Accidentally left the default as full index mode
This PR updates the root cache in the ContractManager when a contract gets renewed, essentially copying over the roots from existing to renewal. I'm keeping this as a DRAFT @n8maninger because it's merely to illustrate the problem, the real fix is (likely) something else entirely and this will also need a thorough regression test. I don't mind taking a stab at it but I felt this is probably something that you will want to write the fix for yourself.
Context:
In renterd we have a test called TestUploadDownloadSpending that fails locally with AppendSector: proof verification failed on the branch pj/subscription-api. The reason it fails is because hostd builds the proof of a recently renewed contract without sector roots. I sprinkled some logging in the host's contract manager and also the proof code in core and found that we seem to update the database just fine on renewal but the next time the contract updater fetches the roots from the store it returns 0 roots. Causing the host to try and build a proof where secRoots is empty causing it to fail when verifying. I considered the possibility that something was removing those roots but I added panics in all delete and trim methods and those aren't hit during the test.
DEBUG PJ: host: contract manager: store renew contract fcid:d0ddaa094b8082ec760af191a241403846b89136ee1b6343f42bda1044a4e434 -> fcid:9fe0027ea0f4e28861fffcc86a06a93969e49acf98fa18d60de7919f89293c7e
DEBUG PJ: store: renew fcid:d0ddaa094b8082ec760af191a241403846b89136ee1b6343f42bda1044a4e434 -> fcid:9fe0027ea0f4e28861fffcc86a06a93969e49acf98fa18d60de7919f89293c7e updated 3 roots
DEBUG PJ: host: contract updater: fcid:9fe0027ea0f4e28861fffcc86a06a93969e49acf98fa18d60de7919f89293c7e append sector, new sector roots cnt 1
DEBUG PJ: host: contract updater: fcid:9fe0027ea0f4e28861fffcc86a06a93969e49acf98fa18d60de7919f89293c7e current sector roots cnt 1
DEBUG PJ: core0.2.3: BuildDiffProof
actions: [{Type:Append A:0 B:0 Data:[]}]
secRoots: []
treeHas: []
leafHas: []
DEBUG PJ: core0.2.3: VerifyDiffProof
actions: [{Type:Append A:0 B:0 Data:[]}]
numleav: 3
treeHas: []
leafHas: []
oldRoot: h:5927721564a242691a0ca9d58f810087ca838fe5205989417296cc6181939067
newRoot: h:c076a802cc2791cb30a710a1977e88e540abff86d091014117ed8e0321c43a66
appRoot: [h:c076a802cc2791cb30a710a1977e88e540abff86d091014117ed8e0321c43a66]
OK: false
Couple of things I find especially odd/interesting is that it passes on CI, so somehow this is a race condition, and also if we really consider the contract to be empty after renewal, even for a short period of time, this would wreak some serious havoc on production so it's probably somehow not that big of an issue or a really small window or something.
Adds "Full" and "None" indexing modes. The indexing mode can only be set once and cannot be changed since it changes how the state tree is stored.
"Partial" is the default. It only stores the state and events for addresses that are connected to a wallet.
"Full" will store the entire state of the blockchain regardless of addresses loaded into the wallet. It will also throw an error if the user attempts to trigger a scan with /wallet/scan
"None" will not process chain updates and treat the chain state as read-only. This isn't particularly useful now, but it will be nice when networked databases are supported.
- Fixed an issue where clearing some configuration fields would crash the app due to how recommendations were being calculated.
- The configuration recommendations now instruct the user to first fill all fields.
- Fixed an issue where the recommendations bar was overlapping notifications.
@peterjan, is this enough to fix the race condition, or do I need to change the deferred release to only be called when the transaction should be dropped?
Bumps go.sia.tech/core from 0.2.2 to 0.2.3.
Commits
f9d44a4Merge pull request #148 from new0nebit/flourishfdfc700Merge pull request #150 from SiaFoundation/surpass54775afMerge pull request #152 from SiaFoundation/dependabot/go_modules/golang.org/x...71ccc09build(deps): bump golang.org/x/crypto from 0.21.0 to 0.22.0c5cb20bImprove SpendPolicy and SatisfiedPolicy tests based on feedback3b09dcbconsensus: Add (State).SufficientlyHeavierThanb038144Add Test Coverage for SatisfiedPolicy and enhance SpendPolicy tests- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/term from 0.19.0 to 0.20.0.
Commits
46c790fgo.mod: update golang.org/x dependencies- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/term from 0.19.0 to 0.20.0.
Commits
46c790fgo.mod: update golang.org/x dependencies- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/sys from 0.19.0 to 0.20.0.
Commits
7d69d98unix: extend support for z/OS7758090cpu: add support for sve2 detection9a28524windows: drop go version tags for unsupported versions27dc90bunix: update to Linux kernel 6.4- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
I'm assuming this snuck in at one point or something. Or maybe this pre-dates us moving ExplicitCoveredFields to coreutils but in any case we should get rid of it because it's not worth having renterd as a dependency (which soon will no longer have a wallet package).
Bumps go.sia.tech/core from 0.2.2 to 0.2.3.
Commits
f9d44a4Merge pull request #148 from new0nebit/flourishfdfc700Merge pull request #150 from SiaFoundation/surpass54775afMerge pull request #152 from SiaFoundation/dependabot/go_modules/golang.org/x...71ccc09build(deps): bump golang.org/x/crypto from 0.21.0 to 0.22.0c5cb20bImprove SpendPolicy and SatisfiedPolicy tests based on feedback3b09dcbconsensus: Add (State).SufficientlyHeavierThanb038144Add Test Coverage for SatisfiedPolicy and enhance SpendPolicy tests- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/sys from 0.19.0 to 0.20.0.
Commits
7d69d98unix: extend support for z/OS7758090cpu: add support for sve2 detection9a28524windows: drop go version tags for unsupported versions27dc90bunix: update to Linux kernel 6.4- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/crypto from 0.22.0 to 0.23.0.
Commits
905d78ago.mod: update golang.org/x dependenciesebb717dssh: validate key type in SSH_MSG_USERAUTH_PK_OK response0da2a6aopenpgp: fix function name in comment5defcc1sha3: fix Sum results for SHAKE functions on s390x- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/term from 0.19.0 to 0.20.0.
Commits
46c790fgo.mod: update golang.org/x dependencies- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/sys from 0.19.0 to 0.20.0.
Commits
7d69d98unix: extend support for z/OS7758090cpu: add support for sve2 detection9a28524windows: drop go version tags for unsupported versions27dc90bunix: update to Linux kernel 6.4- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- 5760861a: Moved the pinned currency change threshold to advanced mode settings.
This changes the wallet.Event JSON serialization to match the Go struct. Unfortunately, this will break the UI again @alexfreska.
- Adds the
maturityHeightfield. The UI should use this to distinguish between received-but-not-spendable payouts (maybe lower the opacity or add a "lock" icon). - Changes the
valfield todatato match the rest ofwalletd.
{ "id": "h:13271ead1070290c9cbe39ed7a15aa1107a27029479858ee9ab2fcc237ed4aa0", "timestamp": "2024-05-01T11:35:03Z", "index": { "height": 68278, "id": "bid:000000009db92b8834aed7809ebe3ff0184e632e8333bba3802de8cd10d2fc86" }, "maturityHeight": 68423, "relevant": [ "addr:11d68c52e67a800d0ca99867e410cfa9c0c3f5bc1726aa90e88f40b7ad0825c91841904dae10" ], "type": "contract payout", "data": {...} }
- Moved the pinned currency change threshold to advanced mode settings.
- Adds a walletd-js library.
import { Walletd } from '@siafoundation/walletd-js' const walletd = Walletd({ api: 'http://localhost:9980/api', password: 'password1337', }) const events = await walletd.walletEvents({ params: { id: 'wallet-1', limit: 10, offset: 0, }, }) const peers = await walletd.syncerPeers() console.log(events.data, peers.data)
- Adds a hostd-js library.
import { Hostd } from '@siafoundation/hostd-js' export async function example() { const hostd = Hostd({ api: 'http://localhost:9980/api', password: 'password1337', }) const state = await hostd.stateHost() const volumes = await hostd.volumes() console.log(state.data, volumes.data) }
It turns out that Chris 's hunch was right and that we are not tracking sector upload estimates at times we should be, even worse we currently don't track anything at times we should be penalising the host for being slow. I added explicit logging for when we penalise a host for being (super) slow, so going to follow this up on my node to be sure.
Overall I thought it was a little tricky to come up with a good way to fix it and to write a solid unit and/or integration test for it. If we really want to check the exact errors we would need an integration test but that would involve a level of orchestration we are currently not capable of doing from an integration test. The unit test I came up with is not exactly what I was hoping for but it does cover all branches.
This implements soft delete for spent elements, sidestepping the issue of reverts clobbering the chain index. Spent elements will be pruned 144 blocks after their spend height. This also reverts the removal of the chain_indices table to use the foreign key as a bug sanity check.
This PR passes an appliedIndex to revertChainUpdate which is used whenever we re-add elements. Before, we were re-adding those elements using the revertIndex which is definitely incorrect because we go and delete those in revertIndex. I renamed revertIndex to removeEvents and changed it to only remove events because if a revert is processed correctly the elements should already have been removed.
The idea of an appliedIndex as the counterpart for every revertIndex feels a bit naive maybe but I decided to propose it nonetheless because I don't see what's wrong with it either. If we can't do that I feel the only thing we're left with is soft deletion but that comes with its own issues, mostly complexity trade-off.
- Fixes a panic when calling
[GET|PUT] /settings/pinnedwith explorer data disabled - Adds explorer info to the
[GET] /state/hostendpoint
2 line change to close statements that were not closed
This PR makes sure that when you run renterd config and select sqlite there's no mysql section in the renterd.yml file (and vice versa of course). I figured this was a good opportunity to make the sqlite database names configurable and refactor the bus config in a way that creating the database connections (and logger) are better placed I think, it was a bit scattered before. There's actually a bunch of small changes that overall clean things up.
Bumps go.sia.tech/gofakes3 from 0.0.2 to 0.0.3.
Commits
d7c039fMerge pull request #6 from SiaFoundation/chris/return-access-keyd01e417move routing into routeBase1f29cb8signature: return access key when V4SignVerify is successful- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps github.com/cloudflare/cloudflare-go from 0.93.0 to 0.94.0.
Release notes
Sourced from github.com/cloudflare/cloudflare-go's releases.
v0.94.0
ENHANCEMENTS:
- access_application: support options_preflight_bypass for access_application (#1790)
- gateway: added ecs_support field to teams_location resource (#1826)
- teams_account: adds custom certificate setting to teams account configuration (#1811)
- workers: support deleting namespaced Workers (#1737)
DEPENDENCIES:
- deps: bumps golang.org/x/net from 0.19.0 to 0.23.0 (#1825)
Changelog
Sourced from github.com/cloudflare/cloudflare-go's changelog.
0.94.0 (April 24th, 2024)
ENHANCEMENTS:
- access_application: support options_preflight_bypass for access_application (#1790)
- gateway: added ecs_support field to teams_location resource (#1826)
- teams_account: adds custom certificate setting to teams account configuration (#1811)
- workers: support deleting namespaced Workers (#1737)
DEPENDENCIES:
- deps: bumps golang.org/x/net from 0.19.0 to 0.23.0 (#1825)
Commits
4cfc90fUpdate CHANGELOG.mde225010update CHANGELOGbdbee54Merge pull request #1826 from alyssamw/masterac539eeMerge branch 'master' into master6f6d756GFI-151: add ecs_support to teams_location5ea2d0aMerge pull request #1825 from cloudflare/dependabot/go_modules/internal/tools...38f5378add CHANGELOG for #1825e96653cbuild(deps): bump golang.org/x/net in /internal/tools04b36cdMerge pull request #1790 from russellkerns/rkerns/AUTH-60007b0666aMerge branch 'master' into rkerns/AUTH-6000- Additional commits viewable in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps github.com/minio/minio-go/v7 from 7.0.69 to 7.0.70.
Release notes
Sourced from github.com/minio/minio-go/v7's releases.
Bugfix Release
What's Changed
- add support for authentication with EKS Pod Identities by
@saweberin minio/minio-go#1944- Allow disabling dual-stack endpoints for Amazon S3 by
@narqoin minio/minio-go#1945- Added
x-minio-replication-actual-object-sizeto allowed headers by@shtripatin minio/minio-go#1946- Update vulncheck Go version to 1.22.x by
@klauspostin minio/minio-go#1953- fix: non md5 etags by
@niger-prequelin minio/minio-go#1952- Bump golang.org/x/net from 0.21.0 to 0.23.0 by
@dependabotin minio/minio-go#1954- Bump golang.org/x/net from 0.21.0 to 0.23.0 in /examples/minio by
@dependabotin minio/minio-go#1955- remove sha256-simd support we do not need it anymore by
@harshavardhanain minio/minio-go#1958New Contributors
@sawebermade their first contribution in minio/minio-go#1944@narqomade their first contribution in minio/minio-go#1945@niger-prequelmade their first contribution in minio/minio-go#1952Full Changelog: https://github.com/minio/minio-go/compare/v7.0.68...v7.0.70
Commits
c2fe179remove sha256-simd support we do not need it anymore (#1958)0e5a768avoid trailing checksums for zero byte objectsafa8d59Bump golang.org/x/net from 0.21.0 to 0.23.0 in /examples/minio (#1955)0558c7eBump golang.org/x/net from 0.21.0 to 0.23.0 (#1954)bfc8151remove broken codeql integration from GitHube44abd3fix: non md5 etags (#1952)18e8073Update vulncheck Go version to 1.22.x (#1953)e8ddcf0Update version to next release- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps gorm.io/gorm from 1.25.9 to 1.25.10.
Commits
9d370bcFix handling of unknown column types (#6540)7892019Fix panic bug in migrator due to lack of nil check for stmt.Schema (#6932)ac59252Add new error for "Violation Check Constraint" (#6992)207f1acfix: not clause with or condition (#6984)85299bfperf: merge nested preload query when using join (#6990)5553ff3downgrade mssql driverbc49365Faster utils.FileWithLineNum (#6981)d0b4cebAdded comment describing Unscoped() method (#6969)9a61ef2fix: duplicated preload (#6948)1e13fd7Fix duplicated columns in INSERT SQL for some fields with default value- Additional commits viewable in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
This PR removes the chain_indices table. I'm not entirely sure that's what we want but since we were discussing getting rid of the CASCADE deletes I don't see a good reason to keep the table. Creating it and linking it is more of a nuisance than anything else imo, if you don't get the upside of the cascaded deletes.
I hit an interesting test failure in the process, an underflow is triggered in TestEphemeralBalance which is interesting because on master there are no orphaned indices, but now there are orphaned siacoin_elements... so I'm a little bit puzzled still on why that is.
I think there might also be a bug in RevertOrphans because we loop over the orphaned indices and it seems we only deal with the balances of the last orphan. (consensus.go:751) Since I'm not sure whether we want to get rid of the chain_indices entirely I'm going to keep it in DRAFT until tomorrow when Nate is back from being OOO.
- Fixed an issue where volume resize and delete dialogs were not loading the volumes data.
- Updated volume resize dialog to used shared helpers for consistency.
Explanation
- Recently fixed an issue where dialogs were not triggering onOpenChange(true) when initially opening, this combined with an incorrect handler that previously never was called caused the
idpassed to volume dialogs to get wiped out.
- 8 min - Moves JS linting, testing, e2e testing out to a parallel job
- 3 min - Moves docker container build and publish out to a parallel job
Our jobs are kind of slow so I was looking at ways to speed them up. I couldn't really find any low hanging fruit but ended up extracting two steps (lint & analyze) I think should be run separately because they do not depend on the os or go-version. It saves us roughly a minute so nothing ground breaking.
I ended up removing the dev trigger for the CodeQL job. I'm fine reverting that change but I feel we run that a lot without any real upside (so far). I can't remember the time it failed and made me change my PR for the better, since it takes more than two minutes on every single run maybe it's fine to only run it when we merge dev.
Caught these errors on renterd shutdown, fixed by adding a context.DeadlineExceeded case in abortRetry.
2024-04-23T10:49:08Z INFO bus successfully saved 392 accounts
2024-04-23T10:49:42Z ERROR SQL trace {"error": "context deadline exceeded", "elapsed": "34.701151966s", "rows": 0, "sql": "\nDELETE\nFROM slabs\nWHERE NOT EXISTS (SELECT 1 FROM slices WHERE slices.db_slab_id = slabs.id)\nAND slabs.db_buffered_slab_id IS NULL\n"}
2024-04-23T10:49:42Z WARN sql transaction attempt 1/7 failed, retry in 200ms, err: context deadline exceeded
2024-04-23T10:49:42Z WARN sql transaction attempt 2/7 failed, retry in 500ms, err: context deadline exceeded
2024-04-23T10:49:43Z WARN sql transaction attempt 3/7 failed, retry in 1s, err: context deadline exceeded
2024-04-23T10:49:44Z WARN sql transaction attempt 4/7 failed, retry in 3s, err: context deadline exceeded
2024-04-23T10:49:47Z WARN sql transaction attempt 5/7 failed, retry in 10s, err: context deadline exceeded
2024-04-23T10:49:57Z WARN sql transaction attempt 6/7 failed, retry in 10s, err: context deadline exceeded
Not doing so caused slight differences in the returned value. e.g. it seemed like the number of usable hosts can improve by manually tweaking settings but no recommendation was returned.
Deployed this to Arequipa for testing @alexfreska
If you set the target to 300 it should give you a recommendation now.
--- Current contracts --- Contracts: 266 --- Target contracts --- Contracts.Amount: 300 --- Before --- MaxDownloadPrice: 1 KS { "hosts": 722, "usable": 273, "unusable": { "blocked": 0, "gouging": { "contract": 12, "download": 109, "gouging": 131, "pruning": 7, "upload": 24 }, "notAcceptingContracts": 83, "notScanned": 86 }, "recommendation": { "gougingSettings": { "maxRPCPrice": "10000000000000000000", "maxContractPrice": "5000000000000000000000000", "maxDownloadPrice": "1155000000000000000000000000", "maxUploadPrice": "333333333000000000000000000", "maxStoragePrice": "154166666667", "hostBlockHeightLeeway": 12, "minPriceTableValidity": 600000000000, "minAccountExpiry": 86400000000000, "minMaxEphemeralAccountBalance": "1000000000000000000000000", "migrationSurchargeMultiplier": 10 } } } --- After --- MaxDownloadPrice: 1.03 KS { "hosts": 722, "usable": 274, "unusable": { "blocked": 0, "gouging": { "contract": 12, "download": 107, "gouging": 131, "pruning": 7, "upload": 24 }, "notAcceptingContracts": 83, "notScanned": 86 }, "recommendation": { "gougingSettings": { "maxRPCPrice": "10000000000000000000", "maxContractPrice": "5000000000000000000000000", "maxDownloadPrice": "1135575000000000000000000000", "maxUploadPrice": "333333333000000000000000000", "maxStoragePrice": "154166666667", "hostBlockHeightLeeway": 12, "minPriceTableValidity": 600000000000, "minAccountExpiry": 86400000000000, "minMaxEphemeralAccountBalance": "1000000000000000000000000", "migrationSurchargeMultiplier": 10 } } }
- Fixed an issue where file health slab keys were showing up as repeated and incorrect.
This removes the random data fill of volumes, enabling Linux and macOS to use sparse files again. However, this does have the side effect of "breaking" size detection when initializing and growing volumes, with no easy solution.I think that's acceptable to reduce hardware wear and tear and increase volume initialization performance.
Another option is to throw this behind an opt-in config flag for advanced users. That would prevent beginners from over-initializing their volumes while allowing advanced users to take advantage of filesystem optimizations.
- Lint passed on windows with both:
git config --global core.autocrlf false.gitattributes:*.go text eol=lf
Switches the nextjs Go router to use path instead of filepath to prevent issues on Windows. Also adds a few sanity tests to make sure the routers initialize correctly.
Fileis only supported in browsers, widening these upload types to accept other data.
- Added support for autopilot config evaluation API to
renterd-jsandrenterd-react. - Added a method for each known bus setting (eg:
bus.settingGouging), in addition to the genericbus.settingmethod.- The hook variants already existed in
renterdand were moved torenterd-react.
- The hook variants already existed in
- Adds proper content type and encoding to file upload and download methods
- Cleans up redundant content type handling code
When a scan is interrupted, orphaned blocks will never be reverted from the wallet store. This adds a check for orphans when scanning and reverts their state from the store.
- Sia Central SDK updated to use
@siafoundation/request. @siafoundation/renterd-jsexamples updated to new API.
I started writing this code during the last hackathon for fun, practicing writing test cases. But I didn't finish it because I had to switch to the main task regarding consensus coverage. I felt bad for wipe out a work that had been done, so I ended up finishing the code and also making some changes to it. Initially, I was only aiming to cover the SatisfiedPolicy but got carried away and started covering the PolicyTypeAbove and PolicyTypePublicKey, completely overlooking the fact that you had already done this 😅 Anyway, here it is. I hope that this alternative approach and the additional scenarios will prove valuable to you 🙏
Bumps github.com/cloudflare/cloudflare-go from 0.92.0 to 0.93.0.
Release notes
Sourced from github.com/cloudflare/cloudflare-go's releases.
v0.93.0
BREAKING CHANGES:
- dns: Remove "locked" flag which is always false (#1618)
ENHANCEMENTS:
- magic_transit_ipsec_tunnel: Adds support for replay_protection boolean flag (#1710)
DEPENDENCIES:
- deps: bumps golang.org/x/net from 0.22.0 to 0.24.0 (#1688)
Changelog
Sourced from github.com/cloudflare/cloudflare-go's changelog.
0.93.0 (April 10th, 2024)
BREAKING CHANGES:
- dns: Remove "locked" flag which is always false (#1618)
ENHANCEMENTS:
- magic_transit_ipsec_tunnel: Adds support for replay_protection boolean flag (#1710)
DEPENDENCIES:
- deps: bumps golang.org/x/net from 0.22.0 to 0.24.0 (#1688)
Commits
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps github.com/shopspring/decimal from 1.3.1 to 1.4.0.
Release notes
Sourced from github.com/shopspring/decimal's releases.
v1.4.0
Full Changelog can be found in CHANGELOG.md
New Contributors
@VadimKulaginmade their first contribution in shopspring/decimal#278@CAEL0made their first contribution in shopspring/decimal#341@zlasdmade their first contribution in shopspring/decimal#301@falsaffamade their first contribution in shopspring/decimal#342@frankli0324made their first contribution in shopspring/decimal#322@cbelsolemade their first contribution in shopspring/decimal#265@acln0made their first contribution in shopspring/decimal#346@davsebymade their first contribution in shopspring/decimal#288@serprexmade their first contribution in shopspring/decimal#355@agmtmade their first contribution in shopspring/decimal#352@richardJiangmade their first contribution in shopspring/decimal#285@axopadyanimade their first contribution in shopspring/decimal#328@Bububugermade their first contribution in shopspring/decimal#131
Changelog
Sourced from github.com/shopspring/decimal's changelog.
Decimal v1.4.0
BREAKING
- Drop support for Go version older than 1.10 #361
FEATURES
- Add implementation of natural logarithm #339 #357
- Add improved implementation of power operation #358
- Add Compare method which forwards calls to Cmp #346
- Add NewFromBigRat constructor #288
- Add NewFromUint64 constructor #352
ENHANCEMENTS
- Migrate to Github Actions #245 #340
- Fix examples for RoundDown, RoundFloor, RoundUp, and RoundCeil #285 #328 #341
- Use Godoc standard to mark deprecated Equals and StringScaled methods #342
- Removed unnecessary min function for RescalePair method #265
- Avoid reallocation of initial slice in MarshalBinary (GobEncode) #355
- Optimize NumDigits method #301 #356
- Optimize BigInt method #359
- Support scanning uint64 #131 #364
- Add docs section with alternative libraries #363
BUGFIXES
Commits
a1bdfc3Release version 1.4.0 (#365)275e48eAdd docs section with alternative libraries (#363)dd603cbSupport scanning uint64 (#364)80ec940Add NewFromUint64 constructor (#352)2fa107dRemove production usage from docs (#362)645a76eOptimize BigInt (#359)ed8f76eDrop support for Go versions older than 1.10 (#361)0e69d5cRun CI on both push to origin branches and pull requests (#360)bf7794eOptimize NumDigits method (#356)547861cAvoid reallocation of initial slice in MarshalBinary (GobEncode) (#355)- Additional commits viewable in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps go.sia.tech/core from 0.2.1 to 0.2.2.
Commits
f424825Merge pull request #151 from SiaFoundation/nate/vzeroupper69d6da0blake2b: clear upper registers for 2x gain1ae0547Merge pull request #149 from SiaFoundation/ifelse097e517lint: Use whitelist for gocritic90599f1lint: Satisfy unslice88ba8c8lint: Disable 'ifElseChain' gocritic checke781eaaUpdate project-add.yml89e9456Merge pull request #147 from SiaFoundation/nate/rhp2-release008e275add release to rhp2 settingsc9bd79dMerge pull request #146 from SiaFoundation/currency-string- Additional commits viewable in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/sys from 0.18.0 to 0.19.0.
Commits
cabba82windows: use uint32 for serial comm flags for consistency1a50d97windows: add serial comm functions95f07ecx/sys/windows: add func windows.DisconnectNamedPipe(handle Handle) (err error)4be02d3unix: expose mmap calls on z/OS- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/crypto from 0.21.0 to 0.22.0.
Commits
d042a39go.mod: update golang.org/x dependenciesb92bf94ssh: respect MaxAuthTries also for "none" auth attempts6f79b5assh: add server side multi-step authentication8d0d405x/crypto/chacha20: cleanup chacha_ppc64le.sb91329dall: remove redundant words in comments and fix some typos- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/term from 0.18.0 to 0.19.0.
Commits
5b15d26go.mod: update golang.org/x dependencies- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/sys from 0.18.0 to 0.19.0.
Commits
cabba82windows: use uint32 for serial comm flags for consistency1a50d97windows: add serial comm functions95f07ecx/sys/windows: add func windows.DisconnectNamedPipe(handle Handle) (err error)4be02d3unix: expose mmap calls on z/OS- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Before
goos: linux
goarch: amd64
pkg: go.sia.tech/core/internal/blake2b
cpu: AMD EPYC 7B13 64-Core Processor
BenchmarkBLAKE2b/SumLeaves-128 4167430 290.8 ns/op 880.38 MB/s 0 B/op 0 allocs/op
BenchmarkBLAKE2b/SumNodes-128 3793945 285.3 ns/op 897.44 MB/s 0 B/op 0 allocs/op
PASS
ok go.sia.tech/core/internal/blake2b 4.610s
After
goos: linux
goarch: amd64
pkg: go.sia.tech/core/internal/blake2b
cpu: AMD EPYC 7B13 64-Core Processor
BenchmarkBLAKE2b/SumLeaves-128 6697410 176.7 ns/op 1448.80 MB/s 0 B/op 0 allocs/op
BenchmarkBLAKE2b/SumNodes-128 5853004 181.6 ns/op 1409.40 MB/s 0 B/op 0 allocs/op
PASS
ok go.sia.tech/core/internal/blake2b 4.376s
- Adds an endpoint for build info to match
hostdandrenterd - Renames
POST /resubscribetoPOST /rescan. This endpoint will now return immediately. - Adds
GET /rescanto track rescan progress - Moves some of the the peer store fields into memory. I noticed that the peer store was causing a lot of write lock contention when syncing. We could solve this by syncing the data less frequently, but the fields didn't seem overly important to keep after the session ends.
Fixes an underflow panic when calling resubscribe
This is mostly in a good place, but there is one major sticking point: wallet.NewManager.
With the old API, when a call to cm.AddBlocks returns, all of the subscribers will have finished processing the new update(s). With the new API, this is not true; the processing happens asynchronously. This broke some tests which expected that, after calling AddBlocks, they could immediately observe the effects of those blocks in wallet state. Now, they can't, so we need a way for the test to wait until processing has finished. In most of the tests, I worked around this; see the syncDB helper I added. But that approach doesn't work when you have a real wallet manager running. For now, I faked the waiting with time.Sleep, just to see if that makes the tests pass (it does), but clearly we need a better solution.
The whole asynchronous processing thing really bugs me, so much so that I tweaked the new API to release the chain manager's lock before calling the onReorg listener functions. This means it's now legal for your listener function to call chain manager methods -- meaning, you can implement subscription "the old fashioned way" and have your onReorg function directly call UpdatesSince in a loop until it's caught up. Effectively this brings us back to the same situation as before, except that the manager is no longer locked, so you have a bit more freedom. (And of course, you can still do async processing if you want.)
But the other big wrinkle here is the wallet manager's Subscribe method. Previously, it unsubscribed from the chain manager, then resubscribed at a lower height. Firstly, it's not clear to me why this would work; won't you end up with duplicate state? But more importantly, we need to nail down how this action ought to interact with the wallet manager's ongoing subscription in the new API. In my PR code, we hold the wallet manager's lock while syncing, so Subscribe effectively pauses the background subscription until it's finished. Not exactly pretty, but it gets the job done. Is there a better way? That's what I'm not sure about.
Old: "~2.529 mS"
New: "2.529378333356156158367 mS"
We discussed removing the SI units as well, and using SC exclusively (i.e. the example above would be 2.529378333356156158367 mS), but there was less consensus on that.
Since String is now exact, ExactString has been removed; if you want raw Hastings values, you can use Big or fmt.Sprintf("%d", c). If you want a rounded value, you can use standard approaches to round the float64 returned by Siacoins.
Bumps golang.org/x/sys from 0.5.0 to 0.18.0.
Commits
360f961unix: add API for fsconfig system call7ff74afunix: drop go version tags for unsupported versions6b4eab5unix: suppress ENOMEM errors from sysctl's implementing Uname(uname *Utsname)...2f2cc5dunix: update IFLA and NETKIT constants with Linux kernel 6.7914b96cwindows: support ill-formed UTF-16 in UTF16PtrToString511ec84Revert "windows: support nil done parameter in ReadFile and WriteFile"628365dwindows: support nil done parameter in ReadFile and WriteFilebef1bd8unix: move mksyscall regexp to package level variables5710a32unix/linux: update Linux kernel to 6.7b3ce6a3windows: build env_windows_test.go only go Go 1.21 and above- Additional commits viewable in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps golang.org/x/crypto from 0.0.0-20220507011949-2cf3adece122 to 0.21.0.
Commits
- See full diff in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
When mining a block with transactions:
- contract formation
- contract revision (broadcast)
- contract revision (broadcast)
- contract revision (broadcast)
I noticed my subscriber receives only 3 file contract elements in the ApplyUpdate, more importantly though one of those did not have a revision. I think there's a bug in reviseFileContractElement that doesn't properly update the rev map when encountering an fce we find in ephemeral, which I assume is a sort of cache.
A mysterious force of nature inspired me to write this code today, thereby increasing the code coverage to 75.7%
The current encoding of Currency uses an 8-byte length prefix followed by a variable-length big-endian encoding of the integer. This is inherently wasteful, as currency values never require more than a 1-byte prefix. Variable-length encodings can also hurt performance in certain use cases, and make it harder to predict the real-world size of various encoded payloads.
This PR adds an alternative fixed-width (16-byte) little-endian encoding for Currency. This matches the actual struct definition of Currency in core: two uint64s. Surprisingly, this is not quite as efficient as simply reducing the 8-byte prefix to a 1-byte prefix; the currency values currently present in the blockchain would encode to between 11-15 bytes on average. Still, the advantages of fixed width outweigh the slightly larger size imo.
As a side effect, I also ended up changing the encoding of SiafundOutput to be more compact in v2.
We ran into bugs related to determining the right amount of collateral when renewing contracts. Most notably exceeding the max collateral setting of hosts.
There were multiple reasons for that. I was able to get rid of most errors by postponing computing the collateral until after we have acquired a contract.
The other issue, which is the reason for this PR, is the fact that we used ContractRenewalCollateral to compute the newCollateral and then CalculateHostPayouts to create the payouts and both of these functions perform the same math but slightly different. (one uses fc.EndHeight() and the other one fc.WindowEnd` for determining the extension of the renewal).
Since we don't use rhpv2 renewals I'm not going to bother refactoring that but I'm updating rhpv3 to make sure there is no duplicate code anymore and that everything, including the capping of the collateral according to the host's MaxCollateral happens in one place.
Right now this is deployed on Arequipa and there are no more collateral errors.
I have been adding this to our repos so we can do things like check struct tags automatically. I know core is still changing so if this is unwanted right now please feel free to close this PR.
Edit: Need to do some Ledger review fixes today. Will fix remaining warnings tomorrow.
I noticed renewal failures in our logs that indicate we're not properly respecting the host's MaxCollateral setting.
{"level":"error","date":"2023-12-07T16:56:27Z","component":"autopilot.contractor","msg":"renewal failed, err: couldn't renew contract: RPCRenew: failed to read RPCRenewContractHostAdditions: ReadResponse: failed to validate renewal: collateral exceeds maximum: expected at most 10000000000000000000000000000 got 10000047475326975980269993984\n","hk":"ed25519:c4b451fa7ab7085763119aeb1aa34db524c5dc6416dc77c5ba82100eb9687790","fcid":"fcid:5a32d8bbfb7e6a29f874ad2ded500d46e3e9aa4675db624b7f782b7fd7f45e05"}
I finished working on these tests last weekend, but due to unforeseen circumstances, I couldn't submit the pull request. Today I can finally do it, but I noticed that just yesterday another contributor, who coincidentally worked on some of the same test features as I did, submitted his PR #132
I wasn't sure how to proceed in this situation, so I decided to run test coverage. In my case, the overall coverage percentage was 77.3%, while @chris124567's was 79.8%, which I congratulate him on. However, if we combine our PRs, the rate reaches 80.1%
Chris and I approached the task and code-writing differently, and to be honest, I'm not sure how you'll handle our PRs after comparing and analyzing them; nonetheless, I decided to offer alternative tests for the Marshaler and Unmarshaler functions and submit this PR.
This is a slightly bigger change, so I figured I'd PR it to allow for some discussion instead of just pushing it.
Sia computes a lot of hashes. When I started implementing v2, I took care to make hashing as fast as possible. In particular, I wanted hashing operations to be zero-alloc. That meant accepting some ugliness like hasherPool and eschewing helper functions that might not be inlined.
This PR essentially reverses that decision. Instead of every ID function grabbing a hasher and directly calling EncodeTo on all the stuff we want to encode, we now pass those things to a magic hashAll helper function. The resulting code is definitely more readable; you can see at a glance exactly what's being hashed. The downside is that all of our hashes now allocate, making them a bit slower. How much slower? Anywhere from 1% to 15%, depending on how little data is being hashed. Is that acceptable? Well, when I run our test suite 100x, it takes basically the same amount of time -- definitely nowhere near 15% slower. Regardless -- preventable allocations make me sad. At least ~all of our hashing will be performed inside hashAll now, which means CPU profiles will show us how much time we spend hashing in general...?
Another consequence is that the exact data we're hashing has changed slightly. Basically, instead of omitting signatures, we now hash empty signatures. Why? Because it means instead of writing this:
func (s State) ContractSigHash(fc types.V2FileContract) types.Hash256 { return hashAll( "sig/filecontract", s.v2ReplayPrefix(), fc.Filesize, fc.FileMerkleRoot, fc.ProofHeight, fc.ExpirationHeight, fc.RenterOutput, fc.HostOutput, fc.MissedHostValue, fc.RenterPublicKey, fc.HostPublicKey, fc.RevisionNumber, ) }
we can write this:
func (s State) ContractSigHash(fc types.V2FileContract) types.Hash256 { fc.RenterSignature, fc.HostSignature = types.Signature{}, types.Signature{} return hashAll("sig/filecontract", s.v2ReplayPrefix(), fc) }
Having a helper like hashAll doesn't force us to make this change, but it sure makes it a lot more compelling, doesn't it?
Lemme know if you have strong feelings about this either way. Frankly, I'm pretty confident that this is the right choice; writing a big post about it is probably just my way of coping with the loss of my precious zero-alloc code 😭
I was working around this in both renterd and hostd but it's easier to update it here instead.
Fixes the node12 deprecation warning
470,562 block height
85,275 online hosts
527 active hosts
7.29 PB total storage
1.70 PB used storage
3,412 M total registry
52 M used registry
20,995 commits
71 contributors
470 forks
43 releases
