stjohn_piano 2021-04-30 yes Problem Summary: While developing blockchain software, how can we cheaply find and fix errors in logic code that executes near expensive blockchain calls? Problem Description: 1) A blockchain application will create blockchain transactions. These transactions, once they settle, are irreversible. They are essentially "signed API calls" to a blockchain. If and when they settle, they will permanently alter that blockchain's state. A blockchain can be thought of as a write-only external API. 2) Inside a blockchain application, there will be important functions (e.g. "createWithdrawal") that do a lot of work - designing, creating, storing, and broadcasting a transaction. 3) Developing new code that will fit into this work sequence is necessary, but also difficult to test. Naively, the entire work sequence is run in order to test the new section of code. Mistakes are expensive and irreversible. 4) Calls to databases can also be expensive. An incorrect database call can create messy data that is very time-consuming to rectify. 5) Calls to external APIs can also be expensive. They may count against a paid monthly limit. And, similarly to database calls, an incorrect API call can create messy data on the company account on the external service that is very time-consuming to rectify. Development Strategy: - We treat all blockchain transactions, database calls, and API calls as "external calls". This list should probably also include "calls to internal APIs or other internal processes", because errors that occur across multiple processes are painful to diagnose, and because clean-up is likely to be time-consuming. - An external call occurs in its own function, which can be tested separately. This is called an "edge function". All edge functions should have timeouts and failure handling. -- Output data can be recorded, and later used as input data when testing other functions. - A function that calls an edge function is called a "blend function". It may call multiple edge functions and / or various logic functions. -- Almost no logic takes place within a blend function. We permit only assignments, function calls, error-handling decisions, and logging actions. -- Even a single syntax error can be expensive to locate and fix if external calls occur during the function. This is why we avoid logic in blend functions (unless the edge functions only retrieve data). It's possible to mock the edge functions, but this can be time-consuming. Putting all logic into separate logic functions for unit-testing is the cheapest way to detect, locate, and fix errors within the logic code, especially during continuous application development. - A function that only performs logic operations is called a "logic function". A function that only calls other logic functions is also a logic function. - Unit tests can be written for logic functions. - End-to-end / behavioural tests can be written for blend functions. -- Mock functions can be written and substituted for any edge functions. Mock functions will return output data that was previously recorded. - Edge functions are tested manually, using a cmd-line tool (logging at DEBUG level). -- The output data can be stored for later use as test input to other functions. -- For database edge functions, it's also possible (although time-consuming) to write tests that spin up a new database, populate it with data, run the edge function, and check that the data in the database is now in the expected final state. -- For blockchain edge functions, it's also possible (although time-consuming) to write tests that spin up a tiny standalone version of the relevant blockchain, populate it with data by executing transactions, run the edge function, and check that the data in the blockchain is now in the expected final state. -- Edge functions that solely retrieve data can be tested with unit tests. These are database SELECT functions, external API GET functions, and blockchain loadTransaction / loadBlock functions. Tactics: - Use argument validation. For crucial functions, this should be very strict. - Use named arguments where possible, not positional ones. This is very helpful when you later add or subtract arguments from functions. - Each module (i.e. file) has its own named logger. This is expensive to set up but extraordinarily helpful for diagnosing problems. - Ideally, loggers are namespaced by filename and path (e.g. they contain something similar to "[appName/chainOps/foo.js]" somewhere in the line) and contain the relevant function name and line number. They are configurable by parent modules (i.e. a parent module can set a child module's log level). Options: Colorised output, an initial timestamp. Log level names (e.g. DEBUG, INFO) should all be padded to the same length. Optionally, the logger logs to a file, with a rolling window (e.g. 30 days). If multiple processes are logging to the same file, use a uniqueID in each log line to differentiate between each output stream. Here's a couple example log lines: 2021-02-02 17:58:11 DEBUG [datafeed_explorer/__init__.py: 38 (setup)] Setup complete. 2021-02-02 17:58:41 INFO [datafeed_explorer/explorer.py: 42 (handle_request)] [request 55397015] New request received. - All external functions return the same thing: A 'result' object containing 'err' and 'msg' properties and various other data properties. -- The data properties depend on the function. They should always be present, even if empty. -- The 'err' property should always be present, even if empty. -- The 'msg' property contains a message string that describes the result of the function. This is helpful for logging. It can be an empty string. -- The 'err' property is an object that contains 'code' and 'msg' properties. --- The 'code' property contains the error code, which permits complex error handling in caller functions. If this is 0, then no error occurred. Always set it to a non-zero value at the start of a function, so that the function only completes successfully if certain conditions are explicitly achieved. --- The err.msg property contains an optional error message, which is helpful for logging and for throwing informative Errors. - At first, we handle errors by throwing them. They are caught and reported in try / catch clauses. -- Over time, check for common errors and return them within the result object. The caller function can then select an appropriate action to take in response to the error. Let's do an example. Here is a NodeJS function called createWithdrawal. This is new code, written for this article as an illustration. It has not been tested. Most of the called functions are not shown. createWithdrawal.js nodejs yes let logger = new Logger({name:'chainOps:withdraw.js', level:'error', timestamp:true}); logger.setLevel('debug'); // tmp: for recording purposes. let log = logger.info; let deb = logger.debug; let jd = JSON.stringify; function lj(foo) { log(jd(foo, null, 2)); } function dj(foo) { deb(jd(foo, null, 2)); } let v = require('validate.js'); // createWithdrawal is a blend function. async function createWithdrawal(args) { try { let err, msg; let name = 'createWithdrawal'; deb(`Entering function: ${name}`) deb(`Args: ${args}`); // Validation functions throw errors if their argument isn't valid. v.validateNArgs({args, n:1}); let {coinSymbol, addresses, transferIDs} = args; v.validateString(coinSymbol); // E.g. "BTC". for (let address of addresses) { v.validateAddress({coinSymbol, address}); } // transferIDs contains the database IDs of these withdrawals. v.validateArray(transferIDs); for (let transferID of transferIDs) { v.validateInteger(transferID); } let finalMsg = ''; let finalErr = {'code': 1, 'msg': '[unknown error]'}; // We set finalErr to contain an error by default. Success must be explicitly toggled. // generateBlockchainTx is a edge function (a blockchain one). let tx; ({err, msg, tx} = await generateBlockchainTx({coinSymbol, addresses})); // Design note: The tx object should contain various properties, including the coinSymbol and the txHex. dj({tx}); // Here, we record the tx data, which can later be used for testing other functions. if (err.code) { logger.error(jd(err)); if (err.code == 1) { // Insufficient funds. let recipient = 'walletSupport'; let report = '[detailed description of problem]'; logger.error(report); ({err, msg} = await sendNotification({recipient, report})); if (err.code) throw Error(err.msg); return {err:finalErr, msg:finalMsg}; } else { throw Error(err.msg); } } // Generate a log message. // This is a simple operation (i.e. create a text string that includes data from an array), but nonetheless we put it within its own logic function, so that it can be unit-tested separately. // This function is shown below the current function. // createLogString4 is a logic function. let logString4; ({err, msg, logString4} = createLogString4({transferIDs})); if (err.code) throw Error(err.msg); log(logString4); // calculateTotalWithdrawAmount is a blend function. ({err, msg, totalWithdrawAmount} = calculateTotalWithdrawAmount({dbConnection, transferIDs}); if (err.code) throw Error(err.msg); lj({totalWithdrawAmount}); // getDBConnection is an edge function. let dbConnection; ({err, msg, dbConnection} = await getDBConnection({})); if (err.code) { finalMsg = `Failed to open dbConnection: ${err.msg}`; deb(finalMsg); // In this case, we return this error in the result object. // The caller function can then handle this particular error in a specific way. return {err:finalErr, msg:finalMsg}; } // beginTransaction is an edge function (or rather, an edge method). ({err, msg} = await dbConnection.beginTransaction()); if (err.code) throw Error(err.msg); // storeBlockchainTx is a (database) edge function. ({err, msg} = await storeBlockchainTx({dbConnection, tx})); if (err.code) throw Error(err.msg); // broadcastBlockchainTransaction is a (blockchain) edge function. let txID; ({err, msg, txID} = await broadcastBlockchainTransaction({tx})); if (err.code) throw Error(err.msg); dj({txID}); // storeBlockchainTxID is a (database) edge function. ({err, msg} = await storeBlockchainTxID({dbConnection, tx, txID}); if (err.code) throw Error(err.msg); // commitTransaction is an edge function (or rather, an edge method). ({err, msg} = await dbConnection.commitTransaction()); if (err.code) throw Error(err.msg); // Explicitly toggle success. finalErr = {'code':0, 'msg':''}; // The creation of this finalMsg value should probably be moved into its own logString5 function, so that it can be easily unit-tested. finalMsg = `${name}: ${coinSymbol} withdrawal transaction created and broadcast. TxID: ${txID}`; return {err:finalErr, msg:finalMsg}; } catch(mainErr) { logger.error(mainErr); ({err, msg} = await dbConnection.rollback()); if (err.code) throw Error(err.msg); ({err, msg} = await dbConnection.release()); if (err.code) throw Error(err.msg); throw Error(mainErr); } function createLogString4({transferIDs}) { let x = transferIDs.join(', '); let result = `Transfer IDs: [${x}]`; return {err:0, msg:'', result}; } // This is a blend function, but it only retrieves data, so we can unit-test it, and it can include logic. async function calculateTotalWithdrawAmount({dbConnection, transferIDs}) { let ids = transferIDs.join(', '); let sql = "SELECT SUM(amount) AS total FROM withdraw WHERE id IN (${ids})"; ({err, msg, result} = await dbConnection.query(sql)); if (err.code) throw Error(err.msg); let totalWithdrawAmount = result.rows[0].total; return {err:0, msg:'', totalWithdrawAmount}; }