Activity

A feed of development activity from across Sia Foundation GitHub repositories.

Dec 8, 2023

DRAFT!

This PR implements webhooks in hostd to address https://github.com/SiaFoundation/hostd/issues/189. It's not done all the way though, but before going too far I want to confirm I'm on the right track. The code here is heavily "borrowed" from renterd, so there's a fair bit of duplication. Is this ok or would it be better to reuse the code from renterd? What would be the preferred method of doing it in that case? Lacking in the version here is the ability to delete webhooks, add actions and cleaning things up a bit.

Dec 7, 2023

Add tagliatelle linter to the golangci-lint configuration to make sure our struct tags follow the goCamel naming convention.

Dec 7, 2023
Dec 7, 2023
  • Contracts now have a details view that shows a graph of the contract's daily funding and spending.
Dec 6, 2023

Closes #227

Dec 6, 2023

We introduced these mutexes to avoid locking issues in the database but have since improved our queries and overall db architecture enough to remove them again. Especially MySQL deployments might see significant performance improvements due to the more granular locking MySQL will use.

Dec 6, 2023

Whether or not a buffer is complete and its size can be inferred from looking at the database upon startup. Meaning that we don't necessarily need to persist these things every time a slab is added.

Dec 6, 2023
Dec 6, 2023

Minor refactor to extract some code into a helper

Dec 6, 2023

Scheduling a trigger when the wallet receives its first deposit makes things really smooth on first setup - so I vote we merge something like this. It essentially replaces blockUntilFunded which I had to remove because we can't block for other reasons (host infos). I should not have removed it but instead replaced it with a trigger. This PR does that.

edit: I don't think it's worth adding a test, it's very NDF prone, I verified it manually

Dec 6, 2023

We have some vulnerabilities in our dependencies. This fixes them.

Dec 6, 2023
Dec 6, 2023

There is an unused contentLength field in s3.UploadPart. On the other hand, when a file is uploaded from the renterd UI, the Content-Length header is always set. Unfortunately, when I try to set the Content-Length header in the API request to the worker, it gets stripped out (probably because of the encoding). For this reason, I set X-Content-Length, which can also be checked by the handler.

Dec 6, 2023

This PR adds pruning metrics, as well as an integration test that covers all metrics we added so far.

Dec 6, 2023

Very big mistake here on this line. Essentially we're caching price tables that weren't registered on the host side. Worst cases I've seen is 2h validity periods so that would mean the host is rendered useless for two hours.

I consider this a hotfix, I'll think about ways to improve it. The mistake was to be anal about having RPCPriceTable returning a hostdb entity... should've just returned it from the start and be safe. The method processing the payment now dictates the Expiry to ensure we'll never hand out "seemingly-valid-price-tables" ever again.

Dec 5, 2023
Dec 5, 2023

@alexfreska this will break a bunch in the UI

Dec 5, 2023
  • Fixed an issue where number fields would not properly handle user input starting with a decimal separator.
  • Fixed an issue with the host interaction values showing as 0s on the host explorer.
  • The advanced configuration now includes a prune sector roots setting.
  • The wallet page now shows the balance evolution again, using the new wallet metrics API.
  • The onboarding wizard no longer shows on the login page.
Dec 5, 2023

Turns out we stopped recording (some) host interactions when we refactored out the interactions table. Found out by running staticcheck by the way, useful tool to look for dead code. I removed all generic Command and ErrorResult stuff since we pass scans and price tables updates separately anyway. I made the worker an InteractionRecorder and attach it to the ctx.

Dec 5, 2023
Dec 4, 2023
Dec 4, 2023

With 1.0 coming we should probably just get rid of these routes to avoid having to mark them as deprecated in 1.1.

Dec 4, 2023

This PR adds a memory manager to downloads similar to the one we have for uploads with 1GB of default memory

Dec 4, 2023
  • Fixed an issue where the filtered contract count was being used in the onboarding checks. Thank you to @multikatt!
  • The file stats overall health now uses the min health reported by the object stats endpoint.
  • Wallet transactions are now paginated.
  • Files can now be sorted by name and health, ascending or descending.
  • File uploads are now disabled if the blockchain is not fully synced.
  • File warnings are now a simple warning icon next to the stats and are explained in a popover.
  • Selecting a file search result now navigates to the directory and applies a file name filter.
  • The bucket empty state now has a button for navigating back to the buckets list.
  • Files in the current directory can now be filtered.
Dec 2, 2023
Dec 2, 2023

Adds a new endpoint to verify a sector's references and check for corruption on disk. This is part 1 of the improvements to host integrity checks.

Dec 2, 2023
Dec 1, 2023
  • Revenue metrics no longer show negative potential on the bar graph.
  • Wallet transactions are now paginated.
  • Earned revenue stat cards are now clearly labeled as earned.
  • Metrics no longer include registry related data.
Dec 1, 2023
  • File uploads are now disabled if the blockchain is not fully synced.
Dec 1, 2023
  • Files in the current directory can now be filtered.
  • Selecting a file search result now navigates to the directory and applies a file name filter.
  • File warnings are now a simple warning icon next to the stats and are explained in a popover.
  • The bucket empty state now has a button for navigating back to the buckets list.
Dec 1, 2023

This PR adds alerts for both upload and download failures.

An open question is how to handle migration related up - and download failures. I haven't added those yet since those errors already bubble up through migration failure alerts. What's unfortunate though is that we lose some context by having the autopilot register those.

Nov 30, 2023

Apparently, the workflow needs to be on the branch to trigger

Nov 30, 2023
Nov 30, 2023

This wasn't worth it and it causes weird behaviour on the hosts page when the wallet isn't funded.

Nov 30, 2023

This prunable data query is rather slow because we have to LEFT JOIN contract_sectors. We need that because we want to include contracts without sectors too. I figured in this case it makes sense to split one query into two: one where we use WHERE NOT EXISTS to catch everything we would miss out on if we were to INNER JOIN.

In the contract size query (so fetching size details for a single contract) I kept the LEFT JOIN since it's bound by the fcid anyway. I considered getting rid of it all together because we only use it in the worker to escape early. We could also return a custom error and return a 400 when the user supplies a contract to prune but there's no sectors to prune. This would remove a bus route though (/contract/:id/size) and I wasn't sure whether that was a good or a bad thing.. might be usable at one point but otoh less routes less trouble... cc @ChrisSchinnerl wdyt.

NOTE: I didn't have to alter the tests because they broke when switching to an INNER JOIN

Nov 30, 2023

This PR uses an update instead of create now when completing a multipart upload. That's possible now since we recently added the object_index to the slices and we no longer rely on the ID for the insertion order.

This also gets rid of the separate PartialSlabs field in the object.Object since workers won't be able to deduct the right download order. Instead partial slabs are now part of the regular slabs but without any slices.

Closes https://github.com/SiaFoundation/renterd/issues/769

Nov 30, 2023
  • Files can now be sorted by name and health, ascending or descending.
Nov 30, 2023

This PR adds a WalletMetric so we can easily fetch the evolution of the balance over time. This would prevent the wallet page from having to load all transactions (80mb) to accomplish the same. This would allow @alexfreska to fetch the latest transactions (using limit=100) and create the balance graph using the metrics endpoint. This is consistent with hostd

	WalletMetric struct {
		Timestamp time.Time `json:"timestamp"`

		Address types.Address `json:"address"`

		Confirmed   types.Currency `json:"confirmed"`
		Spendable   types.Currency `json:"spendable"`
		Unconfirmed types.Currency `json:"unconfirmed"`
	}
Nov 30, 2023
Nov 30, 2023

improves docs

Nov 29, 2023
  • The file stats overall health now uses the min health reported by the object stats endpoint.
Nov 29, 2023
Nov 29, 2023

It seems like pruning slabs is even less straightforward than originally expected. The highly parallel buffer code and append-only files we use for high performance throughput don't play well with pruning. The current code suffers from race conditions between uploading objects and pruning packed slabs which can't easily be fixed. So this PR is reverting that change.

Nov 29, 2023
Nov 29, 2023
  • The send siacoin and siafund features now support specifying custom change and claim addresses.
Nov 29, 2023
Nov 29, 2023
Nov 28, 2023
  • Wallet transactions are now paginated.
Nov 28, 2023
  • Revenue metrics no longer show negative potential on the bar graph.
  • Metrics no longer include registry related data.
  • Earned revenue stat cards are now clearly labeled as earned.
Nov 28, 2023

datasetCount from useContracts() is using the filtered count which makes the onboarding panel appear. Related to https://github.com/SiaFoundation/renterd/issues/765

Nov 28, 2023
  • Refactor types to match latest core changes.
  • Event balances are now calculated with only relevant transaction components.
  • The ledger generate addresses dialog now shows a close action if no new addresses have been generated.
  • Seed wallets now support sending siafunds.
  • Ledger wallets now support sending siafunds.
Nov 27, 2023
Nov 27, 2023

During uploading some backup files with rclone via S3 API, the following error occurred:

ERROR db stores/metadata.go:1493 sql: Scan error on column index 0, name "MIN(sla.health)": converting NULL to float64 is unsupported {"elapsed": "1.358ms", "rows": 1, "sql": "SELECT MIN(sla.health) FROM objects o LEFT JOIN slices sli ON o.id = sli.db_object_id LEFT JOIN slabs sla ON sli.db_slab_id = sla.id WHERE o.id = 4629"}

To the recommendation by @ChrisSchinnerl I have fixed this issue by wrap the MIN() function with COALESCE().

Nov 21, 2023

Adds a sector lock to migration so that sector metadata will not be pruned when it's in the middle of a migration

Nov 20, 2023
  • The revenue graph is now a bar stack, much more clearly showing earned vs potential revenue.
  • The metrics graphs now show a y-axis.
  • Currency display can now be configured to siacoin, fiat, or both along with a preference for when only one can be displayed.
  • App preferences no longer list unused GPU setting.
Nov 20, 2023
Nov 20, 2023
Nov 20, 2023
  • The hostd revenue graph is now a bar stack, much more clearly showing earned vs potential revenue.
  • The metrics graphs now show a y-axis.
  • The renterd contract list now updates after a contract is deleted.
  • The renterd configuration now has zen-specific suggested/default values for number of contract hosts and shard values.
  • The ChartXY API has been refactored.
  • Refactor fonts for library.
Nov 17, 2023
Nov 16, 2023
Nov 15, 2023

Improves new host onboarding by enabling writes to volumes before they finish initializing.

Closes #199

Nov 14, 2023

Hotfix for a bug in the collateral metrics

Nov 13, 2023
Nov 9, 2023
  • Wallet now instructs new users to fund their wallet when there are no transactions yet.
  • The wallet balance tooltip now shows spendable confirmed and unconfirmed values.
  • The configuration now shows suggestions for pricing, collateral, and other settings.
  • Extremely small siacoin values will now show as hastings by default rather than 0SC.
  • Wallet balance evolution graph is now hidden until at least 1 data point is available.
Nov 8, 2023
Nov 8, 2023
Nov 8, 2023
Nov 3, 2023

I finished working on these tests last weekend, but due to unforeseen circumstances, I couldn't submit the pull request. Today I can finally do it, but I noticed that just yesterday another contributor, who coincidentally worked on some of the same test features as I did, submitted his PR #132

I wasn't sure how to proceed in this situation, so I decided to run test coverage. In my case, the overall coverage percentage was 77.3%, while @chris124567's was 79.8%, which I congratulate him on. However, if we combine our PRs, the rate reaches 80.1%

Chris and I approached the task and code-writing differently, and to be honest, I'm not sure how you'll handle our PRs after comparing and analyzing them; nonetheless, I decided to offer alternative tests for the Marshaler and Unmarshaler functions and submit this PR.

Nov 3, 2023

Refers #130

Nov 3, 2023

Get consensus coverage up to 79.8%

Nov 3, 2023
Nov 3, 2023
  • Hosts last scan column now properly displays unscanned values rather than a very large time ago value.
  • Siacoin and number input placeholders now match the suggested value.
  • The autopilot loop is now triggered after settings are successfully updated.
  • Files are now paginated.
  • Removed and updated deprecated route names.
  • Onboarding fund wallet step now requires >0 SC instead of a full allowance.
  • The hosts table now has a last announcement column instead of the known since column.
Nov 1, 2023

Automatically announces the host if their address changes or if they have not announced within the last 180 days. It also adds announcement data to [GET] /state/host

Oct 29, 2023

Added docker example configs for testnet and cleaned up some duplicate stuff

Oct 28, 2023
Oct 27, 2023
Oct 20, 2023
Oct 20, 2023
  • Wallet now shows siafund balances when greater than 0.
Oct 20, 2023
  • The configuration now has an advanced mode that allows the user to view and change all settings.
  • New users will now see an onboarding wizard that prompts the user to complete the necessary setup steps - it also shows the status and progress of each.
  • The configuration page now shows the changed status on fields if the user has made a change but the server values were since updated.
  • The command palette now includes navigation to configuration sections.
  • Fixed an issue where fiat input fields values were not displaying properly.
  • Data tables now refresh themselves without user interaction or refocusing the app, even more frequently if a long running operation is in progress.
  • The configuration is now much simpler by default, only requiring the user to set essential settings.
Oct 16, 2023

This is a slightly bigger change, so I figured I'd PR it to allow for some discussion instead of just pushing it.

Sia computes a lot of hashes. When I started implementing v2, I took care to make hashing as fast as possible. In particular, I wanted hashing operations to be zero-alloc. That meant accepting some ugliness like hasherPool and eschewing helper functions that might not be inlined.

This PR essentially reverses that decision. Instead of every ID function grabbing a hasher and directly calling EncodeTo on all the stuff we want to encode, we now pass those things to a magic hashAll helper function. The resulting code is definitely more readable; you can see at a glance exactly what's being hashed. The downside is that all of our hashes now allocate, making them a bit slower. How much slower? Anywhere from 1% to 15%, depending on how little data is being hashed. Is that acceptable? Well, when I run our test suite 100x, it takes basically the same amount of time -- definitely nowhere near 15% slower. Regardless -- preventable allocations make me sad. At least ~all of our hashing will be performed inside hashAll now, which means CPU profiles will show us how much time we spend hashing in general...?

Another consequence is that the exact data we're hashing has changed slightly. Basically, instead of omitting signatures, we now hash empty signatures. Why? Because it means instead of writing this:

func (s State) ContractSigHash(fc types.V2FileContract) types.Hash256 {
	return hashAll(
		"sig/filecontract",
		s.v2ReplayPrefix(),
		fc.Filesize,
		fc.FileMerkleRoot,
		fc.ProofHeight,
		fc.ExpirationHeight,
		fc.RenterOutput,
		fc.HostOutput,
		fc.MissedHostValue,
		fc.RenterPublicKey,
		fc.HostPublicKey,
		fc.RevisionNumber,
	)
}

we can write this:

func (s State) ContractSigHash(fc types.V2FileContract) types.Hash256 {
	fc.RenterSignature, fc.HostSignature = types.Signature{}, types.Signature{}
	return hashAll("sig/filecontract", s.v2ReplayPrefix(), fc)
}

Having a helper like hashAll doesn't force us to make this change, but it sure makes it a lot more compelling, doesn't it?

Lemme know if you have strong feelings about this either way. Frankly, I'm pretty confident that this is the right choice; writing a big post about it is probably just my way of coping with the loss of my precious zero-alloc code 😭

Oct 13, 2023

This PR updates the core packages. I had to get rid of the (renterd) worker dependency. It causes a cyclic dependency preventing to update the core package in both hostd and renterd:

internal/test/renter.go:16:2: could not import go.sia.tech/renterd/worker (-: # go.sia.tech/renterd/worker
../../../go/pkg/mod/go.sia.tech/renterd@v0.0.0-20230322105544-b8424a111b76/worker/sessionpool.go:19:29: undefined: rhpv2.RPCAppendCost
Oct 13, 2023

Add a test similar to TestP2P in walletd and add some extra cases mostly relating to file contracts to the validation test.

Oct 11, 2023
Oct 11, 2023
Oct 11, 2023

I was working around this in both renterd and hostd but it's easier to update it here instead.

Oct 10, 2023

This PR upgrades the renterd dependency and moves over some session code we removed on renterd's side. This PR is necessary to prepare hostd for upgrading its core dependency.

Oct 5, 2023
Oct 5, 2023
Oct 4, 2023

Fixes numerous issues with migration and sector pruning.

  • Fixes database is locked error while migrating large volumes (closes #179)
  • Fixes an issue with physical sectors not being pruned after being removed from a contract.
  • Adds a migration to recalculate the physical and contract sectors
  • Fixes an edge case causing sector root_index to be off after trimming sectors
  • Adds a migration to reset the root indices
Oct 4, 2023

This PR adds the rpcCost type from hostd. We can move it over to core and get rid of some of the very rough estimates for proof sizes and replace them by exact values we get from DiffProofSize.

Oct 2, 2023

Fixes the node12 deprecation warning

Oct 2, 2023

Increases the go versions used in CI from 1.17 and 1.18 to 1.20 and 1.21

Sep 29, 2023
  • The volume status icon now shows in progress operations more clearly.
  • The contracts table now has a column for "remaining renter funds".
  • Alerts can now be filtered and dismissed by type.
  • The alert dialog now has a fixed height, so it stays in the same position when an alert is dismissed.
  • In-progress volume operations can now be canceled from the volume context menu.
  • Bandwidth metrics are no longer broken out by protocol.
  • Volume operations are now disabled when one is already in progress.
  • The node block height now shows the estimated network height when syncing.
  • Fixed an issue with the alert dismiss button overflowing.
  • The GPU setting now displays the correct device support text.
Sep 29, 2023
Sep 28, 2023

Fixes an underflow panic when trying to spend more than the available budget.

Sep 22, 2023

Signs Windows releases to satisfy Windows smart screen.

Sep 20, 2023

Exposes active RHP sessions over the API

Sep 7, 2023

This removes a few random whitespaces, and fixes the build script for docker so it works on forked repos. It'll still obviously be the same for this repo too.

Sep 6, 2023
  • Wallet seeds now get cached after an address is successfully generated.
  • Ledger and seed wallets should now use the correct fee when sending transactions with or without the include fee option.
  • walletd now supports Ledger-based wallets.
  • Ledger wallets now support generating a specific address or a range of addresses via Ledger device.
  • Outputs are now properly released when a transaction fails during signing or broadcast.
  • The GPU setting now displays the correct device support text.
  • Fixed an issue with the syncing status staying at 99%.
  • Addresses can now be deleted from the context menu.
  • Both ledger and seed wallets now save public keys in address metadata, this is especially important to streamline Ledger signing.
  • Support for connecting Ledger devices over USB, HID or Bluetooth depending on browser support.
Aug 29, 2023

Adds validation tests for V2Blocks/V2Transactions.

Aug 25, 2023
Aug 10, 2023

Not sure if this is the right place to put this considering that the return type is a boolean, but some kind of safeguard against this seems appropriate.

Aug 9, 2023
  • The app now warns the user if it is running on the testnet with a prominent banner.
  • Table headers now freeze in view when the table scrolls.
  • Fixed an issue where walletd would indefinitely report as "syncing" - the synced state is now based on the most recent block's timestamp being within the last 2 hours.
  • The contrast was improved on the syncing progress text in the daemon profile.
Aug 7, 2023

Prevent an unnecessary 1 KB allocation from being repeated in all the put calls.

Jul 22, 2023

Changes boltDB interface in node.go to match changes in https://github.com/SiaFoundation/core/pull/121

Jul 19, 2023

This began as an effort to make the DB commit less frequently, which is important for speeding up IBD. Currently, we commit (i.e. fsync) after every block, which means hundreds of thousands of fsyncs are required for a full IBD -- often many fsyncs per second. Needless to say, this is enormously wasteful. Ideally, we would fsync once every few seconds, and/or after buffering a large number of writes. So that's what this PR does: it tracks how long it's been since the last fsync, and how many writes we've buffered since then, and only flushes after we cross a threshold.

In the course of implementing this, I realized that our DB interface was more powerful than necessary. Specifically, it supports fully-isolated transactions, meaning you can have multiple "views" of the DB open concurrently, and those views won't observe each others effects, and can be committed or rolled back independently. However, our usage of the DB is already serialized by the chain.Manager's mutex, and clients of chain.Manager only have limited access to the DB. So we actually don't need transactions; all we need is a way to force an fsync, so that we can guarantee that a subscriber will never commit data that hasn't been committed by the manager itself. That's what this PR does.

This still needs to be tested and benchmarked on an actual node, so I'll hold off on merging until then. Hopefully we'll see a significant speedup!

Jul 18, 2023
  • Currency options now include AUD.
  • The wallet list now has a column that shows the lock status. If the wallet is unlocked, the tooltip also shows the remaining time.
  • The wallet list now has a siacoin balance column.
  • Wallet dropdown now includes unlock, edit, and delete operations.
  • Wallet auto-locking can be enabled or disabled from the App Preferences menu - the feature is enabled by default. The locking inactivity period can also be configured.
  • Seed-based wallets can now be unlocked for a period of time. Each wallet will lock after a period of inactivity or if the app is closed.
  • The Settings dialog is now called App preferences.
  • Operations that require a seed, such as sending siacoin or generated addresses now use cached seed values when a wallet is already unlocked.
  • The app now includes an auto-lock feature that can be enabled or disabled from the App Preferences menu. The locking inactivity period can also be configured.
  • The login screen now has an option to show the custom API field.
  • The login screen timeout for an unresponsive daemon is now 10 seconds.
Jul 1, 2023
Jun 30, 2023
Jun 27, 2023
May 30, 2023

hostKey is passed to PrepareContractRenewal, although it is not used there.

May 29, 2023
May 25, 2023

To avoid accidentally dialing localhost in case the port is set but the address isn't.

May 5, 2023
// MarshalText implements encoding.TextMarshaler.
func (uk UnlockKey) MarshalText() ([]byte, error) {
	return marshalHex(uk.Algorithm.String(), uk.Key[:])
}

Causes UnlockKey to JSON encode as "\"ed25519\":48a468d62e48e27ef24707bd1f467476e9605954e201834c659e614b9704d041"

This will likely break a bunch of things in hostd and renterd. Is adding compatibility code for this in core worth it?

Apr 28, 2023
Apr 28, 2023

This PR adds the InitProgramHelper which makes it a bit nicer to compute costs for programs.

e.g. computing the cost of a read sector program becomes this

rc := pt.InitProgramCost()
rc.Add(pt.ReadSectorCost(rhpv2.SectorSize))
cost, _ := rc.Total()

where previously we couldn't make use of the ResourceCost type for the base cost.

Also snuck in a receiver change that Nate proposed.

Apr 20, 2023

This should prevent an overflow panic when the hosts set their prices ridiculously high. Could fix renterd/#283

Apr 18, 2023
Apr 18, 2023

This came up in a renterd review the other day so I decided to propose it. I'm still not sure whether it's silly or actually useful so I figured I would let you be the judge Luke. Definitely feel free to just close this out, I was expecting way more occurrences to be honest, was surprised to only find a handful.

Apr 18, 2023

This PR adds a Name field to the Network object in the consensus package. By adding the network name the UI can differentiate between the mainnet and the testnet and adjust accordingly.

Apr 13, 2023

Turns out if you have a MarshalText method, you don't need MarshalJSON too. So that's nice.

I still want to write a JSON encoder at some point; it would be really nice if we could autogenerate fast encode/decode functions, with options like useCamelCase and noNullSlices -- but for now, struct tags will have to do.

Apr 13, 2023

ok, I did something kinda weird here: I implemented a txpool, but I did it inside the chain manager.

This requires some justification. The "normal" way to implement the txpool would be as a subscriber to the chain manager. This is how it's done in siad: modules.TransactionPool subscribes to modules.ConsensusSet. The pool receives transactions from the user (and from peers), and when a block comes in, it removes any confirmed transactions from the pool. Simple, right? This is the approach I initially used for core as well.

Once I had finished prototyping and started hooking everything up, though, I ran into some difficulties. I came up with workarounds, but something still felt off. The final straw came when, after implementing (what I thought was) the final workaround, I realized that I had introduced a deadlock. It was time to take a larger step back.

The source of the trouble was transaction validation. The txpool has to ensure that all of the transactions in the pool could potentially be included in the next block. So when a new block is mined, all of the transactions in the pool have to be revalidated against the new state. Transactions in the pool may also depend on each other, so invalidating one transaction could cascade to many more. Separately, validation also has to occur when new transactions are added to the pool. (These requirements are what led me to implementing #107) But here's the problem: validating transactions requires access to the consensus database. This creates a dangerous cycle: the consensus database is owned by the chain manager, so the txpool needs to call into the chain manager to validate new transactions; but the txpool is also subscribed to the chain manager, so the chain manager needs to call into the txpool to give it new blocks. The situation is ripe for a deadlock.

siad solves this problem by having the txpool call a special method on the chain manager: LockedTryTransactionSet. This is a function that takes a function that takes a transaction validation function and calls it while holding the chain manager mutex, so that the second function can call the validation function arbitrarily while protected by the lock. Simple, right? 🙃 (I was, um, not particularly happy with this solution, and expressed as much in the commit message and branch name.)

I think I must have repressed the memory of writing that code, because otherwise I wouldn't have plowed ahead with the same exact architecture in core. It was only when things failed to come together nicely that I remembered what had transpired years ago...

ppf-lol

Anyway, when you have two components that are all mixed up in each other's biz, there's really only one solution: smash them together into one big component. Usually this feels wrong; my gut insists that the chain manager and txpool are "different things" that shouldn't live under one roof, and there's a way to cleanly separate them if I try hard enough. But hey, sometimes your gut is wrong. After all, the deadlock is gone now -- can't argue with that.

Interestingly, this problem entirely disappears in Utreexo. There's no more consensus database, so the txpool no longer needs to access it through the chain manager; instead, it can maintain its own copy of the full consensus state (updated via subscription), and validate all transactions against that -- nice! So once we migrate to Utreexo, it should be possible to split these two components apart again. But who knows, there may be other reasons to keep them together that aren't clear yet. Time will tell.

Oh, one more tidbit: the new txpool is lazy. Rather than revalidating the pool after every block, it only does so when necessary, e.g. when someone calls PoolTransactions. A similar principle is applied to the RecommendedFee and UnconfirmedParents methods. In steady-state, this probably doesn't make a big difference in performance (since the UI will poll more than once per 10 minutes), but when you're syncing thousands of blocks, it absolutely does!

Apr 8, 2023

This is an unfortunately-messy diff, but the gist is that consensus now has a MidState type that tracks the state of the blockchain within a block.

When we verify or apply a transaction, we do so relative to the prior block's state and the effects of all previous transactions in the current block. For example, if transaction 0 spends an input, we don't want to allow transaction 1 to spend it too, even though both spends are valid in the context of just the prior block's state.

siad handles this by directly applying the effects of each transaction to its consensus.db, rolling back if an error is encountered. But in core, we want to cleanly separate validation from application; the Validate functions do not have write-access to the consensus database. Thus, any tracking of mid-block effects has to be handled in memory. Previously, this was done on an ad-hoc basis. For example, validateSiacoins iterated over a list of transactions, inserting the relevant parent IDs into a map[types.SiacoinOutputID]int, while validateContracts performed a similar operation with a map[types.FileContractID]types.FileContract. The new MidState type essentially collects all of these maps into one place, so that the validation functions can focus on, well, validation, rather than updating state. This also results in some nice unification of code that was duplicated across the validate functions and ApplyDiff.

The actual motivation for this change, however, came from the txpool. Part of the txpool's job is to produce a list of transactions that could potentially form a block; therefore, it naturally needs to do the same sort of mid-state validation/application as consensus. Furthermore, for efficiency, we really want to store this mid-state somehow, rather than having to recompute it every time a new transaction is submitted to the txpool. It all seems obvious in retrospect, but I had to stumble through a few different approaches to this problem before finally identifying "mid-state" as the unifying concept. :)

I have re-synced a core node with these changes and it validated to tip without any problems, so I'm pretty confident in its correctness. That said, we'll want to do a final pass on all consensus code before deploying anything on mainnet.

Apr 5, 2023

Right now RPCRenewSignatures doesn't match RPCRenewContractRenterSignatures and RPCRenewContractHostSignatures from siad. So rhpv3 renewals fail due to a decoding error.

Apr 4, 2023

Adds the same helpers we already have for rhpv2 but slightly updated for rhpv3.

NOTE: taxAdjustedPayout is a copy-paste and exactly the same for rhpv2. Feel free to suggest a better place for it or to push a commit that moves it.

Apr 1, 2023

This is a broad refactor that removes hot-wallet functionality from walletd. The new plan is for walletd to be a watch-only server, like walrus. Address generation and transaction signing will be performed by clients, namely the built-in UI.

This PR also includes a syncer implementation. This is a core piece of any Sia node, and will (eventually) be copied into renterd and hostd. It also includes a txpool, but really just the stub of one, not a proper implementation. This is the last piece of functionality needed to fully migrate to core, so it's my top priority right now.

So, why push incomplete code? Simple: it's so that @alexfreska can look at the API endpoints and start working on a UI. 😉

Mar 24, 2023
Mar 21, 2023
Feb 28, 2023

Delete wallet/bip39.go. Missed this yesterday but it's not necessary anymore since this logic is in core now.

Feb 27, 2023

Use the seed functions, StandardAddress, StandardUnlockConditions from core/wallet and remove the copies in wallet/types.go

Feb 9, 2023

Still need to switch out the consensus diff stuff in walletutil (current source of build failures) but that shouldn't be too hard.

Jan 19, 2023
Jan 16, 2023

Add this action to the workflow to check API route consistency with https://github.com/SiaFoundation/jape.

Dec 14, 2022

Mostly implements the algorithm described at https://bitcoin.stackexchange.com/questions/1077/what-is-the-coin-selection-algorithm. Still need to add tests.

Oct 28, 2022

Port https://github.com/lukechampine/us/blob/master/wallet/builder.go#L128 from us and add a corresponding API endpoint.

Oct 11, 2022

Adds a JSON store and corresponding tests to preserve wallet data.

Oct 5, 2022

Essentially copies the test from wallet/hotwallet_test.go but retrieves data using the HTTP walletd api instead of directly through the Wallet interface.

Oct 4, 2022

Adjusts walletd's API to use the jape framework instead of httprouter.

Oct 1, 2022

Add tests in the style of https://github.com/lukechampine/us/blob/master/wallet/seedwallet_test.go to walletd. Required a little adapting. Adding tests to make sure the client/server are doing what they are supposed to be doing would probably also be useful.

Sep 14, 2022

Add /wallet/transaction and /wallet/siacoins in accordance with the specification laid out at https://api.sia.tech/.

Aug 30, 2022

Adds basic ephemeral wallet information store and ProcessConsensusChange implementation.