geekAbyte

FuturesOrdered: Keeping Concurrency in Order In Rust

noreply@blogger.com (dade) — Sun, 14 Apr 2024 17:56:00 +0000

The use case is as follows: You have some tasks you want to execute concurrently, but you want the results to be returned in the same order the tasks were defined to be executed.

As an example, let's say you have some records returned from the database, as specified by some query. But before returning the records, some transformation needs to be done, and these transformations can be done concurrently. Due to the nature of concurrency, the order in which the transformations will be done is nondeterministic, meaning the order of the transformed records can and will end up being different from the original order of the records as retrieved from the database. This is something you do not want; you want the transformed records to be in the same order as specified in the original query that retrieved them from the database.

This sounds like the perfect task for Rust's FuturesOrdered type from the futures crate.

Reading the documentation, it states:

"...it imposes a FIFO order on top of the set of futures. While futures in the set will race to completion in parallel, results will only be returned in the order their originating futures were added to the queue."

Just what we want.

Let us illustrate with some code.

The dependencies:

[dependencies]
futures = "0.3.30"
tokio = { version = "1.37.0", features = ["full"]}
rand = "0.8.5"

The code:

async fn transform(number: i32) -> i32 {
    let mut rng = rand::thread_rng();
    let random_number = rng.gen_range(1..=5);
    tokio::time::sleep(Duration::from_secs(random_number)).await;
    println!("Done from {number}");
    return number;
}

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let mut tasks = FuturesOrdered::new();
    for n in 0..9 {
        tasks.push_back(async move { transform(n).await })
    }
    let mut results = vec![];
    while let Some(result) = tasks.next().await {
        results.push(result)
    }
    println!("{results:?}");
    Ok(())
}

The transform function represents the concurrent task. It sleeps for a random amount of seconds between 1 and 5, prints it is done, and then returns the transformed value, which in this case is just the original value passed to the function.

Running the code above should give something like:

Done from 1
Done from 3
Done from 6
Done from 9
Done from 5
Done from 7
Done from 0
Done from 2
Done from 4
Done from 8
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

The Done from n shows that the tasks are finished in different ordering, but the final printing of the results is still in order, showing that even though the tasks get executed concurrently and finishes out of order, when the result is retrieved via calling `tasks.next().await` it is returned in other.

Another way is to collect the results into a vec. That is:

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let mut tasks = FuturesOrdered::new();
    for n in 0..=9 {
        tasks.push_back(transform(n))
    }
    let results = tasks.collect::<Vec<i32>>().await;
    println!("{results:?}");
    Ok(())
}

With this, the tasks would also execute, concurrently, finish in nondeterministic order, but the results will be collected in order as the tasks were started.

Fixing "Failed to get Recent Blockhash" Error in Solana Development with Anchor Framework

noreply@blogger.com (dade) — Fri, 05 May 2023 08:39:00 +0000

I took a break from exploring development in Solana, and the first thing I hit when I recently resumed again was failing tests when using the Anchor framework.

Running anchor run test leads to this error:

  1) calculator
  Is initialized!:
  Error: failed to get recent blockhash: FetchError:
  request to http://localhost:8899/ failed, reason:
  connect ECONNREFUSED ::1:8899
  at Connection.getLatestBlockhash
  (node_modules/@solana/web3.js/src/connection.ts:4555:13)
  at processTicksAndRejections
  (node:internal/process/task_queues:95:5)
  at AnchorProvider.sendAndConfirm
  (node_modules/@coral-xyz/anchor/src/provider.ts:147:9)
  at MethodsBuilder.rpc [as _rpcFn]
  (node_modules/@coral-xyz/anchor/src/program/namespace/rpc.ts:29:16)

The important part of the error message is:

Error: failed to get recent blockhash: FetchError:
request to http://localhost:8899/ failed, reason:
connect ECONNREFUSED ::1:8899
      at Connection.getLatestBlockhash

Googling the error, I found this suggestion here which says to "Try using node 16 instead of 17 if you're on 17".

I tried this and sure this works. But it does not feel right. node's current version is 20.x, dropping down to version 16 does not feel like the most appropriate way to solve this problem.

So why is the issue happening in the first place? Maybe if we understand that, we can come up with a better solution.

Well, the reason why dropping down to version 16 works, is because, starting from version 17, node now by default resolves "localhost" to IPv6's ::1, not IPv4's 127.0.0.1. Version 16 was the last version were node resolves localhost to 127.0.0.1.

And you guessed it, when you run the local Solana test validator, via solana-test-validator it is automatically listening on 127.0.0.1. So Anchor test is trying to connect to the validator using "localhost", but this resolves to an IPv6 address, but our Solana validator is listening for connections on an IPv4 address, which is 127.0.0.1

Now that we know the problem, a better solution then is to configure the Solana test validator to listen on the IPv6 address, or better still configure it to listen on 0.0.0.0 which means it will be able to serve request that comes in both as IPv4 or IPv6.

To do this, stop your Solana test validator if it was running and then run the following command:

solana config set --url http://0.0.0.0:8899
// or to set to IPv6
solana config set --url http://\[::1\]:8899

Start your test validator and run the tests, everything should work fine.

A Short Note on Types in Rust

noreply@blogger.com (dade) — Mon, 23 Jan 2023 07:41:00 +0000

Types, loosely defined, are about capabilities. What are the capabilities available on a value of a certain type? Programmers experienced with strong and statically typed languages will probably have a more visceral appreciation of this, but even programmers of weakly and dynamically typed languages like JavaScript, encounter the reality of types and their capabilities; only that they do so at runtime.

For example, a JavaScript developer might type out the following code:

> "3".toExponential()

But this will fail at runtime because the "3" is of the String type and it does not have the capability to be shown as exponential. The following, on the other hand, will work seamlessly:

> (3).toExponential()
'3e+0'

Because 3 is of the number type and the number type has the capability to be formatted in exponential form.

Rust extends the idea of types to now include capabilities that relate to memory management. These capabilities ensure memory safety in Rust.

A short note on memory

Dealing with memory while programming is an area that is mostly hidden from developers who have mostly worked with languages with garbage collectors. This section quickly provides a brief overview of some of the key aspects of memory management, as moving to Rust will require a deeper understanding of what is happening behind the scenes.

Stack and Heap

Values within a program take up memory. There are various memory regions in a computer, where values can be placed, but two of the most common regions are the stack and heap.

You can think of the stack as a temporary memory location for values needed by an executing function. This includes function parameters and locally defined variables. Since this is more of a scratch area, access to it is fast, and once a function is done executing, the values can be discarded/overridden. Hence, the typical lifespan of a value on the stack is tied to the function that needs it for execution.

The heap, on the other hand, is a memory location that is more durable, where values that are not tied to the lifetime of an executing function can be placed. Dealing with the heap is more involved for obvious reasons. The heap is not as simple as the stack, and finding space to put new values might require additional low level memory management operation needed to make such space available. The stack does not have this complexity.

The idea is to limit the use of the heap region as much as possible, as it is not as fast as the stack memory region.

Sized Types and Dynamically Sized Types

The type of a variable determines the type of values such a variable can contain. A consequence of this is that types also dictate how big or small the memory required by such a value is.

A variable of type u32 can contain numeric values in the range of 0 up to 4,294,967,295, while a variable of type u8, on the other hand, has a smaller size and can contain numeric values between the range of 0 and 255. This means a variable of u32, will require a memory that is 32-bit long, while u8, requires memory that is 8-bit in length.

Most types have a particular size, hence required memory length, that is knowable at compile time. u8 and u32 fall into this category of types. These types are called Sized Types. They are guaranteed to remain uniquely the same. A type of u32 will always need a 32-bit length of memory regardless if it contains 0 or 4,294,967,295. The same applies to all other sized types.

On the other hand, there are other types whose sizes cannot be uniquely known at compile time.

One example is an array represented by [T]. This type represents a certain number of T in sequence, but we don’t know how many there are; it could be 0, 1, 132, or 1 million T's. Hence, it is not possible to ascribe a unique size to these types at compile time. These types are known as Dynamically sized types (DST).

The str type is another example of DSTs. It represents a slice of strings. But for the same reason why we cannot uniquely determine the size of [T] at compile time, we also cannot determine the size of all slices of strings. Because a slice of string, represented by the type str could be 0, 1, 131, or 1 million long.

The idea with DSTs is not that the size is not known at compile times; they are, but they can vary, hence no unique size can be ascribed to them at compile time.

The Rust compiler generally prefers to know the size of types at compile time, for various reasons such as better management and optimization. So, we have a problem here with dynamically sized types, since creating a value and annotating it with a DST won't compile.

This:

fn main() {
   let dst_value: str = "hello world";
}

Will fail compilation with the following error

error[E0277]: the size for values of type `str` cannot be known at compilation time
  --> src/main.rs:70:9
   |
70 |     let dst_value: str = "hello world";
   |         ^^^^^^^^^ doesn't have a size known at compile-time
   |
   = help: the trait `Sized` is not implemented for `str`
   = note: all local variables must have a statically known size
   = help: unsized locals are gated as an unstable feature

The Rust compiler is kind enough to tell us why it is refusing to compile the code. A very instructive reason it gave is that "all local variables must have a statically known size."

So, how do we deal with this?

To fully understand how we first take a look at the concepts of ownership and the borrow checker in Rust.

Ownership, Copying, Referencing, Moving and Borrowing

Let's first look at copying and referencing. We can illustrate using Javascript.

Copying

This is one of the atomic operations performed in any programming language. You have value in one variable, you copy it into another. In Javascript:

> let a = 1
undefined
> let b = a
undefined
> b
1
> b = 2
2
> b
2
> a
1

Basically, once you have b = a the value of a is copied to b, hence when b is modified, it does not affect the value still contained in a.

Referencing

Referencing is when multiple variables do not have their unique copy of a value, but instead point to the same value. This means that if one of the variables changes the value, the other variable will also be changed.

> let a = [1,2,3]
undefined
> let b = a
undefined
> a.push(4)
4
> a
[ 1, 2, 3, 4 ]
> b
[ 1, 2, 3, 4 ]

To prevent creating a reference, a clone will have to be created.

The concepts of copying and referencing also exist in Rust, but Rust handles them differently. Before discussing copying in Rust, let's take a look at the concepts of ownership and moving of values, which on the other hand, are unique to Rust.

Ownership and moving of values

Rust introduces the concept of ownership and the moving of values. Simply put, it means values are not assigned to variables, rather values are owned by variables. For example:

>> let a = 1;

Should now be seen as the variable a was granted ownership of the value 1.

It is also possible to copy a value owned by a variable, which creates another value that is owned by the other variable.

>> let a = 1;
>> let b = a;
>> a
1
>> b
1

This should be interpreted as variable a owning the value 1, then with let b = a, a new copy of the value owned by the variable a is created and handed over to be owned by variable b.

The above is more or less the same behavior as the copying we saw with JavaScript.

The additional feature in Rust is the concept of moving values.

This means, instead of copying, a variable yields ownership of a value to another variable. As a result, once the value is moved and ownership is transferred, the old variable can no longer be used.

fn main() {
   let a = String::from("Hello world");
   let b = a; // value has moved from a to b
   println!("{b}");
   println!("{a}"); // this line won't make it compile
}

The above code won't compile. This is because with the values of type String, the let b = a line means the value from variable a is moved to variable b. That is, variable a has transferred ownership to variable b. Hence, trying to use the variable a later on in the code by attempting to print it out won't compile.

What could be confusing is the fact that in the previous example where we worked with values of type u8, the behavior was copying, which is similar to what we had with JavaScript. But, when the value is of type String, the value was moved!

What gives?

Well, it comes down to the idea of types again and their capabilities. The general idea is, types allocated on the heap generally default to moving values, while most primitive types that are placed on the stack copy instead. String is a type that manages memory on the heap hence why its default capability is not to copy. To create a distinct copy of String it will have to be cloned instead.

It is illustrative to point out that Rust further distinguishes between copying and cloning. Where copy can be interpreted as a simple bit-wise transfer of bits from one memory location to another while clone, apart from moving bits, also require additional logic to be executed.

This whole concept of moving values is something unique to Rust and new programmers coming to Rust need to be aware of this.

Borrowing and referencing of values

Now to recreate the scenario we had in Javascript where mutating an array in one variable led to the mutation on another variable, we have the following Rust code:

fn main() {
   let mut a = vec![1,2,3,4];
   let b = &mut a;
   b.push(5);
   println!("{a:?}");
}

Variable a holds a vector, but it was mutated via variable b and 5 was added, and this got reflected in variable a. Essentially, the same reference scenario we had with JavaScript arrays.

The only difference, however, is that Rust makes referencing explicit.

Let's go through the syntax. let mut a = vec![1,2,3,4]; grants ownership of the vector to variable a. Most data structures are immutable by default in Rust, so we use the mut keyword to indicate that the value owned by a can be modified.

Then let b =&mut a; is where the magic happens.

This, instead of moving the value from a to b, allows b to borrow the value instead.

This form of borrowing is achieved by creating a reference and handing that over to b.

The syntax &mut is used to achieve this, and the mut keyword indicates that it is possible to mutate the value through this reference.

It is possible to borrow a value, i.e. create a reference to a value but not be able to mutate the value. Sort of a read-only. In such a scenario, you only use & for example:

fn main() {
   let mut a = vec![1,2,3,4];
   let b = &a;
   println!("{b:?}");
}

But there are certain things you can and cannot do depending on if a value is owned, moved, borrowed, mutable, or immutable. These whole classes of restrictions are there to ensure memory safety, and this is what the borrow checker enforces.

In summary, the borrow checker ensures: you can have multiple read-only references to the same memory location as long as there are no mutable references. If a mutable reference is created to a memory location, it is only this mutable reference that can read from and write to the memory location.

To get a more elaborate exploration of the rules the borrow checker enforces, see Rust Ownership Rules

Introduction to Pointers and using References to work with DST

DSTs were already introduced as types that do not have a unique size at compile time, and as mentioned above, Rust prefers to deal with types it can know their size at compile time. So, how does Rust work with DSTs? To understand this, we need to talk about pointers.

Pointers and References in Rust

Pointers are variables, just like any other type of variable, but their values hold the memory address of other variables. They are an indirection that points to the memory location where another value can be found.

In Rust, pointers also have types, which makes sense since one can imagine, specific capabilities are only available to pointers. Pointers can also be found in languages like C and C++, but they can be difficult, and unsafe to work with due to the potential for misusing the unhindered access to memory locations.

This is the reason in Rust, you hardly deal with pointers directly. Instead, you have references, which are pointers with safety or liveness guarantees.

You can think of them as a protective layer that makes working with pointers safer. Pointers that are not references, i.e. that don't have this protective guarantee in Rust are usually called raw pointers.

The thing with references is that they also have types, and the good thing about them, is that they have a known size. This is because references hold memory addresses, hence it is possible to have a constant bit length that will always be assigned to references, big enough to enable it to store whatever memory address it needs to store.

Sometimes extra meta-information about the value in the memory address they point to is also stored, when this is the case, the reference is usually called a fat pointer.

The syntax for creating references is &T. To create a reference to a type T, use &T. For example, in the following code:

use std::collections::HashMap;
fn main() {
   let phone_code: HashMap<String, u8> = HashMap::from([("NL".to_string(), 31), ("USA".to_string(), 1)]);
   let ref_to_phone_code: &HashMap<String, u8> = &phone_code;
   dbg!(ref_to_phone_code);
}

phone_code is of the type HashMap<String, u8>, but ref_to_phone_code, a reference, is of the type &HashMap<String, u8> - notice the ampersand now in the type.

Also, the value of the type &HashMap<String, u8> was created by appending & to the variable it will reference, that is &phone_code in the code above.

Rust uses references to work with DSTs. Since references are types with known size at compile time, the trick is to only allow interaction with DSTs via references to them. This is why annotating a type with str, a DST, will cause a compilation failure, but &str, a reference to a DST, compiles fine.

In Summary: T (Sized and DST), &T and &mut T

As mentioned at the beginning of this post, types are about encoding capabilities allowed with a value. Rust extends this to encompass capabilities related to memory management and layout.

So a type T can exist in one of two flavors: One where its memory size is always known to be of a particular bit length; these are called Sized Types, and the ones whose size/bit length is not unique and can vary; these are called Dynamically Sized Types.

Then you have references, which are pointers (variables that hold memory location) with guarantees that ensure safe memory access.

These guarantees are enforced by the borrow checker.

These references could be of type &T or &mut T, depending on whether the reference is mutable or not. If type T is a DST, then a variable of type &T (or &mut T) can be used to reference the DST.

Additional Resources

Introduction to Declarative Macros in Rust

noreply@blogger.com (dade) — Tue, 11 Oct 2022 17:11:00 +0000

This post will give a quick overview of declarative macros in Rust. If you are not familiar with the concepts of macros in Rust, then read Introduction to Macros in Rust first.

Declarative macros work similar to how pattern matching works. It involves taking inputs, then pattern matching to determine what logic to execute.

The macros definition for creating declarative macros, then contains rules which determine the Rust code that gets generated/executed based on the pattern.

This is why declarative macros are also sometimes referred to as Macros By Example, as the code that is generated depends on the example of patterns that matches.

Rust provides the macro_rules! macro for defining declarative macros. You use the macro_rules! to specify the input string pattern and also to specify the code that gets generated/executed.

The syntax for macro_rules takes the following form:

macro_rules! $name {
    ($matcher) => {$expansion} ;
    ($matcher) => {$expansion} ;
    // …
   ($matcher) => {$expansion} ;
}

That is, the macro_rules definition contains a list of rules, the particular rule, represented by {$expression} that will be executed depends on which of the ($matcher) matches the input. To see this syntax in action, create a new cargo project by running

cargo new declarative_macros

Open the main.rs and edit it.

The $matcher can be a simple literal match. For example to create a macro that prints different message based on whether the input passed to it is "morning" or "evening" , add the code below to main.rs:

fn main() {

    #[macro_export]
    macro_rules! greet {
        ("morning") => {
            println!("Good morning! Have a nice day!");
        };
        ("evening") => {
            println!("Good evening! Sleep tight!");
        }
    }
}
// using it
#[test]
fn run_greet() {
    // prints "Good morning! Have a nice day!"
    greet!("morning"); 
    // prints "Good evening! Sleep tight!"
    greet!("evening");
}

The $match can be more versatile than just matching a literal. For example the match can be used to define that the input should be an expression, a block of code etc. It can also be used to capture values passed in, that are then used within the macro logic.

To demonstrate how using a more versatile match pattern looks like when defining declarative macros, we write one that retrieves the balances for specified Solana accounts.

For this to work, update the dependencies section in Cargo.toml as follows:

[dependencies]
solana-client = "1.8.1, < 1.11"
solana-program = "1.8.1, < 1.11"

Then update the main.rs file as follows:

use solana_client::rpc_client::RpcClient;
use solana_program::pubkey::Pubkey;
use std::str::FromStr;

fn main() {
 #[macro_export]
 macro_rules! sol_balance {
  ($item:expr) => {
    {
     let mut result: Vec<u64> = Vec::new();
     let rpc = RpcClient::new("https://api.devnet.solana.com"
               .to_string());
     let item_str = format!("{}", $item);
     for i in item_str.split(",") {
         result.push(rpc.get_account(
         &Pubkey::from_str(i.trim()).unwrap()
         ).unwrap().lamports);
       }
       result
     }
  }}

}

#[test]
fn run_macros() {
    let result = sol_balance!(
        "4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU,
        HEvSKofvBgfaexv23kMabbYqxasxU3mQ4ibBMEmJWHny"
    );
    dbg!(result);
}

Running the test would print out the sol balance at the specified account. Let's go over the code above. The core structure of the macro is:

    #[macro_export]
    macro_rules! sol_balance {
        ($item:expr) => {
        // body of macro
    }}

The ($item:expr) is an example of a more versatile match. It still corresponds to the ($matcher) in ($matcher) => {$expansion}. The $item is the input that is being matched, but this time around it is not just a literal. This syntax also captures the input making its value available for use in the macro code. This is called Metavariables. It also contains a qualification part using :expr. This is used to categorize the input. The :expr is an example of a fragment specifier. :expr specifies that the input is an expression. Other fragment specifiers exist. See Fragment Specifiers for more information.

The body of the macro is then defined as:

{
  let mut result: Vec&lt;u64&gt; = Vec::new();
  let rpc = RpcClient::new("https://api.devnet.solana.com"
            .to_string());
  let item_str = format!("{}", $item);
  for i in item_str.split(",") {
      result.push(rpc.get_account(
      &amp;Pubkey::from_str(i.trim()).unwrap()
      ).unwrap().lamports);
     }
  result
}

Note that we wrap this within {} to allow using multiple statements.

It is also possible to support repetition when defining the matcher. This take the form of:

$ ( $(matcher) ) separator repmode

You can read this as $(matcher) as the pattern to be repeated, the separator that delineates the repetition, and repmode determines the number of repetition that is expected.

The currently supported repmode are:

?: at most one repetition *: zero or more repetitions +: one or more repetitions

The same syntax is used to process each item that is repeated. To see this repetition in practice, we can update the sol_balance macro to take a list of accounts separated by ";" instead of taking a single string separated by ",". This will look like:

use solana_client::rpc_client::RpcClient;
use solana_program::pubkey::Pubkey;
use std::str::FromStr;

fn main() {

 #[macro_export]
 macro_rules! sol_balance {
   ($($item:expr);*) => {
     {
       let mut result: Vec<u64> = Vec::new();

       let rpc = RpcClient::new("https://api.devnet.solana.com"
                 .to_string());
         $(
           let item_str = format!("{}", $item);
           result.push(rpc.get_account(
           &Pubkey::from_str(item_str.trim()).unwrap()
           ).unwrap().lamports);
          )*
          result
       }
    }}
}

// usage

#[test]
fn run_macros() {
  let result = sol_balance!(
    "4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU"
     ;
     "HEvSKofvBgfaexv23kMabbYqxasxU3mQ4ibBMEmJWHny"
   );
  dbg!(result);
}

Running the test with the code modified as above should also print the sol balance at the given accounts.

There is more to writing a declarative macros that is not touched upon in this overview. For example we did not talk about the #[macro_export] attribute, neither did we talk about things like Metavariable Expressions. Even at this, this introduction should be enough to understand the essence of declarative Macros in Rust.

The next post in this series will then be an introduction to the function-like procedural macros.

Introduction to Macros In Rust

noreply@blogger.com (dade) — Mon, 10 Oct 2022 12:38:00 +0000

This guide will present an overview of the fundamentals a new developer approaching macros in Rust needs to know. If you have read other materials online and you still find the concept of macros in Rust vague, then this post is for you.

The guide will be broken into 4 series of posts. The first post, which is this post, will be a quick overview of macros in Rust. The following 3 posts will then look more deeply into the different types of macros in Rust and how to create them.

The posts in this series are:

Introduction to macros in Rust (this post)
Introduction to declarative macros in Rust
Introduction to attribute-like procedural macros in Rust (yet to be published)
Introduction to function-like procedural macros in Rust (yet to be published)
Introduction to custom-derive procedural macros in Rust (yet to be published)

What are macros

Macros are mechanisms for generating or manipulating code programmatically at compile time. In essence, it allows us to treat source code as values. Meaning that, at compile time, we can take inputs: where the input could be another source code or some defined string pattern, and from these inputs, manipulate and generate source codes that form the final compiled code that will be executed.

Macros as a concept are not specific to Rust, as they can be found in languages like Lisp, C/C++, Scala, and a host of others. Although one clear distinction of macros in Rust is that it provides type safety. Meaning that it provides mechanisms to prevent generating programs or source code that are not type-safe.

That is what macros are in a nutshell, a mechanism for taking inputs (other source codes, or just string patterns) and using these to generate source code/program at compile time.

So now that we know what macros are, we talk about how macro usage looks like.

What does macros usage look like in Rust?

Before looking at how to create macros in Rust, it is instructive to see how already created macros are used.

There are two distinct ways of making use of macros in Rust. macros are either used via A syntax I call bang function syntax or via attribute syntax.

The bang function syntax is the syntax that looks like a function call but with an exclamation mark at the end of the function name. For example things like: println!(), dbg!(), format!().

These are used to invoke macros and are not normal function calls.

Attributes syntax, on the other hand, is the syntax used for annotation. It takes the form of #[..] or #![..] and they can be placed on various language elements like functions, structs, enums, etc. This is another syntax that can be used to invoke macros.

A real-world example of attribute syntax being used to invoke macros can be found in the web framework called rocket. For example in the code below, gotten from the documentation here where we see the definition of a handler for the / HTTP path:

#[get("/")]
fn index() -> &'static str {
    "Hello, world!"
}

The #[get("/")] syntax is invoking a macro defined by the rocket framework. This invocation means that at compile time, the custom get macro will take the function index() as input, process it, and generate the exact source code, the framework needs to be able to call this function in response to an HTTP GET request to the path /. Essentially fulfilling what a macro is: a mechanism to generate source code at compile time.

The exact syntax used to call a macro depends on the type of macro and how it has been created. Some macro types are called via the bang function syntax, while other kinds are called via the attribute syntax.

Types of Rust macros.

In Rust, there are two types of macros: Declarative macros and Procedural macros. There are then 3 subcategories of procedural macros: Custom-derive macros, Attribute-like macros, and Function-like macros.

Basically types of Rust macros:

Declarative macros (also referred to as macro by example or mbe for short)
Procedural macros
- Function-like
- Attribute-like macros
- Custom derive macros

The first thing to note in this categorisation is what makes declarative macros different from procedural Macros.

The difference lies in their inputs and outputs, and also how they are created and used.

Declarative macros take what can be described as token fragments while procedural macros take TokenStream as inputs. Token fragments are like string patterns that can further be qualified by a fragment specifier that tells what kind they are. Fragment specifiers can be used to specify the inputs are block, expression, statement, etc. See Fragment Specifiers for an overview of the supported fragment specifiers.

Procedural macros, on the other hand, take in TokenStream which is a more low-level representation of the source code.

The other difference between declarative macros and procedural macros (also the difference between the 3 kinds of procedural macros) is how they are created and used.

Declarative macros and function-like procedural macros are used by a syntax that looks like function invocation but ends with !. for example do_work!(). Meaning when looking at usage it is not possible to tell if a declarative macro is being used or a function-like procedural macro is.

Attribute-like procedural macros on the other hand are used to create a custom annotation, so they are used with the syntax of applying attributes. That is: #[my_macro]. They can also be defined to take helper attributes. That is #[my_macro(attr)]

While custom derives macros are used to define new values that can be passed into the #[derive(...)] macro. For example, if MyMacro is defined to be a custom derive macro, it can then be used as #[derive(MyMacro)]

This completes the quick overview of macros in Rust. The following posts in the series will then go into the details of how these different types of macros are created and used and their different peculiarities.

How to query account balance on Solana using Rust.

noreply@blogger.com (dade) — Wed, 28 Sep 2022 03:22:00 +0000

This post shows how to query Solana's cluster rpc endpoint using Rust. Most of the examples of how to perform this task that I ran to uses Javascript/Typescript via the Solana Web3js library, so I decided to do a quick post showing how to do same but in Rust. It uses querying for the sol balance of a Solana account as the main example of interaction with the rpc endpoint.

The task is simple: given a Solana address, we retrieve the sol balance for that address.

To get started we will need the following dependencies:

[dependencies]
solana-client = "1.14.3" // update to the current latest version
solana-sdk = "1.14.3" // update to the current latest version

The code snippet below then shows how to call the rpc endpoint and retrieve the Sol balance for an account

use solana_client::rpc_client::RpcClient;
use solana_program::pubkey::Pubkey;
use std::str::FromStr;

fn main() {
 let rpc = RpcClient::new("https://api.devnet.solana.com");
 let pubkey_str = "4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU";

 let balance = rpc
      .get_account(&Pubkey::from_str(pubkey_str).unwrap())
      .unwrap().lamports;

 println!("Sol balance of {pubkey_str} is {balance}");
}

Solana addresses are base58 encoded strings, hence they need to be first converted into an instance of Pubkey before they can be used in the rpc.get_account call, which expects a &[u8] byte array. To help with this conversion, Pubkey implements the std::str::FromStr trait. This wmeans, if the trait std::str::FromStr is in scope, then it is possible to convert a string to Pubkey, which is what was done in the code snippet above.

This post was inspired by a recent question I answered on Solana stack exchange here. The answer also mentions another verbose alternative that involves manually converting from base58 to bytes array.

IntoIterator and the for ... in Syntax in Rust

noreply@blogger.com (dade) — Sat, 13 Aug 2022 17:41:00 +0000

In Rust Iterator pattern with iter(), into_iter() and iter_mut() methods I explained why attempting to use a variable holding a Vec after iterating through it using the for … in syntax leads to a compilation error.

The post explains why the following code won't compile:

fn main() {
   let some_ints = vec![1,2,3,4,5];
  // iterating through a vec
   for i in some_ints {
       dbg!(i);
   }
// attempting to use the vec will 
// lead to compile error after the iteration
   dbg!(some_ints);
}

I then showed 3 methods that can be called before iterating using the for … in and how 2 of these methods allow the Vec to still be used even after iteration.

These 3 methods are into_iter(), iter(), and iter_mut(). That is:

#[test]
fn into_iter_demo() {
    // the .into_iter() method creates an iterator, v1_iter 
    // which takes ownership of the values being iterated.
    let mut v1_iter = v1.into_iter();

    assert_eq!(v1_iter.next(), Some(1));
    assert_eq!(v1_iter.next(), Some(2));
    assert_eq!(v1_iter.next(), Some(3));
    assert_eq!(v1_iter.next(), None);

    // If the line below is uncommented, the code won't compile anymore
    // this is because, after the iteration, v1 can no longer be used 
    // since the iteration moved ownership
    //dbg!(v1);
}

The two other methods that allow the Vec to still be used after iteration via for … in are:

#[test]
fn iter_demo() {
    let v1 = vec![1, 2, 3];
    // the .iter() method creates an iterator, 
    // v1_iter which borrows value immutably 
    let mut v1_iter = v1.iter();

    // iter() returns an iterator of slices.
    assert_eq!(v1_iter.next(), Some(&1));
    assert_eq!(v1_iter.next(), Some(&2));
    assert_eq!(v1_iter.next(), Some(&3));
    assert_eq!(v1_iter.next(), None);
   // because values were borrowed immutably, 
   // it is still possible to use 
   // the vec after iteration is done
    dbg!(v1);
}

And

#[test]
fn iter_mut_demo() {
    let mut v1 = vec![1, 2, 3];

    // the .iter_mut() method creates an iterator, 
    // v1_iter which borrows value and can mutate it. 
    let mut v1_iter = v1.iter_mut();

    // access the first item and multiple it by 2
    let item1 = v1_iter.next().unwrap();
    *item1 = *item1 * 2;

    // access the second item and multiple it by 2
    let item2 = v1_iter.next().unwrap();
    *item2 = *item2 * 2;

    // access the third item and multiple it by 2
    let item3 = v1_iter.next().unwrap();
    *item3 = *item3 * 2;

    // end of the iteration
    assert_eq!(v1_iter.next(), None);

    // this will print out [2,4,6]
    dbg!(v1);
}

In this post, we are going to dive a little bit deeper into understanding some of the machinery that makes the above work.

We start again by talking about the Iterator trait.

Iterator pattern and Iterator trait.

An Iterator represents the ability to retrieve elements from another data structure in sequence. In rust, it is any data structure that implements the Iterator trait.

It is important to note that the Vec data structure by itself is not an Iterator and hence cannot be iterated.

To make this more obvious, let us forget about the for … in syntax for a second, and try to perform an operation that should be possible on a data structure that supports being iterated.

An example of such an operation is the for_each method.

In the code example below, we attempt to loop directly over a Vec of numbers using for_each and print each item. The code won't compile:

fn main() {
   let some_ints = vec![1,2,3];
   // calling for_each directly on a Vec won't compile
   some_ints.for_each(|item| {
       dbg!(item);
   });
}

The compile error below gives us a clue as to why the code does not compile:

error[E0599]: `Vec<{integer}>` is not an iterator
  --> src/main.rs:60:14
   |
60 |      some_ints.for_each(|item| {
   |                ^^^^^^^^ `Vec<{integer}>` is not an iterator; try calling `.into_iter()` or `.iter()`
   |
   = note: the following trait bounds were not satisfied:
           `Vec<{integer}>: Iterator`
           which is required by `&mut Vec<{integer}>: Iterator`
           `[{integer}]: Iterator`
           which is required by `&mut [{integer}]: Iterator`

For more information about this error, try `rustc --explain E0599`.
error: could not compile `playground` due to a previous error

The compile error contains the line:

Vec<{integer}> is not an iterator; try calling .into_iter() or .iter()

Proving the point that a data structure like Vec by itself is not an iterator.

But instead of using the for_each method, we can actually perform an iteration directly on the Vec using the for … in syntax.

For example, the following code compile and runs as expected:

fn main() {
   let some_ints = vec![1,2,3];
   for item in some_ints {
       dbg!(item);
   }
}

What gives?!

Did we not just prove that a Vec is not an Iterator on itself?

We even showed this by trying to call a method that should work on an iterator and confirm it fails. But here we are still being able to iterate over something that should not be an Iterator using the for … in syntax.

How is that possible?

To understand why this works, we need to look into another trait called IntoIterator.

What is the IntoIterator trait?

The IntoIterator is a trait that specifies how a data structure can be converted into an Iterator. The basic structure of the trait looks like this:

pub trait IntoIterator {
    type Item;
    type IntoIter: Iterator
    where
        <Self::IntoIter as Iterator>::Item == Self::Item;

    fn into_iter(self) -> Self::IntoIter;
}

As can be seen, the main method specified by the trait is into_iter(). The result of calling it with is an Iterator. That is, Self::IntoIter the return type, is of type Iterator, given it is an associated type defined to be an iterator in the body of the trait. This is what the line type IntoIter: Iterator above means.

So any data structure, that by itself is not an iterator, can define how it can be transformed into an iterator by implementing the IntoIterator.

The Vec data structure defined in the Rust standard library implements the IntoIterator, which means it has the method into_iter() which when called returns an Iterator.

To see this in action, let's go back to the code that did not compile, where we directly called for_each on a Vec, but instead of calling for_each directly, we first call into_iter(), before calling for_each.

fn main() {
   let some_ints = vec![1,2,3];
   // first calling into_iter() works
   some_ints.into_iter().for_each(|item| {
       dbg!(item);
   });
}

This works, because the first call to into_iter() returns an Iterator, which allows iterating over the underlying Vec.

So how does this help answer why it is possible to use for … in directly on a Vec without first turning it into an Iterator by calling into_iter?

Well, the answer is that when the for … in syntax is used, the compiler automagically first calls into_iter(), getting an Iterator back and using that to do the iteration.

According to the documentation, the for … in syntax actually desugars to something like this:

let values = vec![1, 2, 3, 4, 5];
{
    let result = match IntoIterator::into_iter(values) {
        mut iter => loop {
            let next;
            match iter.next() {
                Some(val) => next = val,
                None => break,
            };
            let x = next;
            let () = { println!("{x}"); };
        },
    };
    result
}

Where the into_iter is first called on the Vec value, and then the iteration is continuously done, calling next(), until None is reached, signifying the end of the iteration.

So the into_iter method, which is part of the IntoInterator traits explains how the for … in syntax can be used for iterating over a Vec. And this is because the compiler by default calls the into_iter when for … in syntax is used.

But what about the other two similar methods that we saw at the beginning of this post? That is iter() and iter_mut().

It is also possible to use these two methods to turn a Vec into an iterator.

That is:

fn main() {

   let some_ints = vec![1,2,3];
   // first calling into_iter() works
   some_ints.iter().for_each(|item| {
       dbg!(item);
   });
}

and

fn main() {
   let mut some_ints = vec![1,2,3];
   some_ints.iter_mut().for_each(|item| {
       dbg!(item);
   });
}

What is going on when iter_mut() and iter() are used? And how is this different from the into_iter() that comes from the IntoIterator trait?

3 different kinds of iteration

Iterators can come in different forms. Nothing is stopping a developer from implementing an Iterator that has other custom behavior that defines how it iterates.

In Rust's standard library, most collections have 3 different kinds of Iterators. We can have one which takes ownership of the value being iterated, one that borrows the value immutably, and another that borrows the value and can mutate it.

An iterator that takes ownership can be created by calling into_iter(), one that borrows immutable can be created by calling iter() and the one that borrows value with the ability to mutate can be created by calling iter_mut(). This is the crux of the Rust Iterator pattern with iter(), into_iter() and iter_mut() methods post.

It turns out that the Rust compiler by default goes for the into_iter() version when it de-sugars the for … in syntax.

One important thing to point out here is the fact that it is possible to have a custom data structure, that is an iterator, i.e. has all the familiar iteration-related methods: map, for_each etc but which is not usable with the for … in syntax. This will be the case if such data structure implements Iterator but does not implement the IntoIterator trait. Because without implementing the IntoIterator trait, there will be no into_iter() method for the for … in syntax to call.

Another interesting point is what happens if we manually call any of into_iter(), iter(), or iter_mut ourselves as part of usage in for … in syntax. Basically what was shown in the Rust Iterator pattern with iter(), into_iter() and iter_mut() methods post.

How come these works:

fn main() {
   let mut some_ints = vec![1,2,3];
// manually calling iter in a for … in 
   for i in some_ints.iter() { 
       dbg!(i);
   }

// manually calling iter_mut in a for … in 
   for i in some_ints.iter_mut() {
       dbg!(i);
   }

// manually calling into_iter in a for … in    
   for i in some_ints.into_iter() {
       dbg!(i);
   }
}

We are manually converting the Vec ourselves to an iterator by calling iter(), iter_mut(), and into_iter() and yet it works.

Why does this work?

Was it not already stated that the for … in syntax works with anything that implements IntoIterator which allows it to call into_iter(). And here we are, manually converting the Vec into an iterator ourselves, and yet the for … in works. How come?

The answer is in a little trick in the standard library. Which is the fact that the standard library contains this implementation for IntoIterator:

impl<I: Iterator> IntoIterator for I

This basically means any Iterator implements IntoIterator and the implementation is such that the Iterator returns itself when into_iter() is called. Which makes sense if you think about it. If something is already an Iterator, what else can be done when you attempt to turn it again into an Iterator other than it returning itself?

And this is what happens. Even though the iter(), into_iter() and iter_mut methods are called directly, for … in, still work, because the iterator created by calling this method automatically has an implementation of IntoIterator which returns itself and which the for … in syntax needs.

The above shows how implementing the IntoIterator trait can be done in such a way as to provide interesting functionalities.

Another interesting utility that is achieved via providing different implementations for IntoIterator is how it is possible to use for … in with a collection and yet be able to still use the collection after iteration, without having to call the iter() or iter_mut.

This is a more succinct syntax to the solution provided in the Rust Iterator pattern with iter(), into_iter() and iter_mut() methods post

We look at this, in the next section.

The 3 Implementation of IntoIterator for Vec

There are three different implementations of IntoIterator for the Vec type. These 3 different implementations are for 3 different variants of the Vec type depending on its memory access.

There are implementations of IntoIterator for the bare Vec<T> type, the immutable referenced &Vec<T> type and mutable reference &'a mut Vec<T> type.

The implementation of IntoIterator for bare Vec<T> returns an Iterator that takes ownership of the values as they are iterated. The implementation of IntoIterator for &Vec<T> borrows the value being iterated immutable, while the implementation of IntoIterator for &'a mut Vec<T> makes it possible to mutate the value as part of the iteration.

This means one can iterate over a Vec type and still be able to use it afterward if the iteration is done over &Vec<T> or &'a mut Vec<T>. For example:

fn main() {
   let some_ints = vec![1,2,3];
   for i in &some_ints { // same as calling some_ints.iter()
       dbg!(i);
   }
}

and

fn main() {
   let mut some_ints = vec![1,2,3];
   for i in &mut some_ints { // same as calling some_ints.iter_mut()
       *i = *i * 2;
   }
   dbg!(some_ints);
}

The above syntax can be used as a more succinct way to iterate over a data structure like Vec using the for … in syntax without taking ownership of the Vec.

Summary

The IntoIterator is a trait that defines how an Iterator can be created for a data structure. It defines an into_iter() method that when it is called, should return an Iterator
The for .. in syntax requires there is an implementation of IntoIterator because the compiler automagically first calls into_iter() to retrieve an Iterator it uses for its iteration.
The Rust standard library also contains an implementation of IntoIterator for an Iterator. This implementation just returns the Iterator. This makes sense, because if something is already an iterator, then returning it, satisfies the contract defined by IntoIterator
The fact that there is an IntoIterator for Iterator that returns that iterator means that methods like iter() or iter_mut() can still be used within the for .. in syntax. The for .. in syntax, can call the into_iter() gets the Iterator itself and uses that for its iteration.
The standard library also contains 3 different implementations of IntoIterator for Vec<T>, &Vec<T>, and &'a mut Vec<T>. The implementation for Vec<T> takes ownership of the value being iterated, the implementation for &Vec<T> borrows values being iterated immutably, and the implementation for &'a mut Vec<T> borrows values mutably.
Given a variable some_var holding a Vec one can iterate over it using for … in and still be able to use it after the iteration if the iteration is done over &some_var and &mut some_ints.

Rust Iterator pattern with iter(), into_iter() and iter_mut() methods

noreply@blogger.com (dade) — Wed, 10 Aug 2022 21:37:00 +0000

Let's create a vec of integers, iterate through and print the individual values, and then afterward, print out the whole vec. Here is the code:

fn main() {
   let some_ints = vec![1,2,3,4,5];
   for i in some_ints {
       dbg!(i);
   }

   dbg!(some_ints);
}

Simple right? Well nope. Trying to compile the above code will fail with the following errors:

error[E0382]: use of moved value: `some_ints`
   --> src/main.rs:9:9
    |
4   |    let some_ints = vec![1,2,3,4,5];
    |        --------- move occurs because `some_ints` has type `Vec<i32>`, 
    which does not implement the `Copy` trait
5   |    for i in some_ints {
    |             --------- `some_ints` moved due to this implicit call to `.into_iter()`
...
9   |    dbg!(some_ints);
    |         ^^^^^^^^^ value used here after move
    |
note: this function takes ownership of the receiver `self`, which moves `some_ints`
help: consider iterating over a slice of the `Vec<i32>`'s 
content to avoid moving into the `for` loop
    |
5   |    for i in &some_ints {
    |             +

For more information about this error, try `rustc --explain E0382`

Why is this the case? Why is the borrow checker preventing the use of a vec after a simple iteration?

Well, the answer to that question lies in Rust's implementation of the Iterator pattern - which by the way, is what makes it possible to use the for…in syntax.

Iterators are not special or unique to Rust. The concept can be found in a handful of languages. I wrote about the Iterator pattern as it exists in JavaScript in the post Iterables and Iterators in JavaScript.

The unique thing about the Iterator pattern in Rust is its interaction with the borrow checker.

If this interaction with the borrow checker is not taken into consideration then it is possible to bump into certain confusing compile errors while attempting to use the iterator pattern.

So to get started answering the question of why the borrow checker prevents what looks like a legit code above, let's take a look at the Iterator pattern and what is special about it in Rust.

Rust and Iterator Pattern.

An Iterator is a data structure that allows retrieval of elements from another data structure in sequence. In rust, it is any data structure that implements the Iterator trait.

#![allow(unused)]
fn main() {
pub trait Iterator {
    type Item;

    fn next(&mut self) -> Option<Self::Item>;

    // methods with default implementations elided
 }
}

The Iterator trait defines a next method that when called, returns an Option of type Self::Item, which ends up being whatever the type of elements in what is being iterated over. Once the iteration reaches the end, a None is returned. For example:

#[test]
fn iterator_demo() {
 let v1 = vec![1, 2, 3];
 let mut v1_iter = v1.iter();

 assert_eq!(v1_iter.next(), Some(&1));
 assert_eq!(v1_iter.next(), Some(&2));
 assert_eq!(v1_iter.next(), Some(&3));
 assert_eq!(v1_iter.next(), None);

}

The above is simple enough. The part that interacts with the borrow checker is how the next method treats the value it is getting from the underlying structure it is iterating over.

It is possible that the next method of the iterator trait be implemented in such a way that it borrows the value immutable, borrows the value mutable, or even takes ownership of the values being iterated.

These three behaviors define 3 different distinct types of iterators.

An iterator that borrows the value being iterated immutably
An iterator that borrows the value being iterated mutably.
An iterator that takes ownership of values being iterated

To demonstrate these 3 iterator types, we can take a vec as usual and create these three distinct iterators and observe their behavior.

An iterator that borrows the value being iterated immutably.

#[test]
fn iter_demo() {
    let v1 = vec![1, 2, 3];
    // the .iter() method creates an iterator, v1_iter 
    // which borrows value immutably 
    let mut v1_iter = v1.iter();

    // iter() returns an iterator of slices.
    assert_eq!(v1_iter.next(), Some(&1));
    assert_eq!(v1_iter.next(), Some(&2));
    assert_eq!(v1_iter.next(), Some(&3));
    assert_eq!(v1_iter.next(), None);
   // because values were borrowed immutably, it is still 
   // possible to use the vec after iteration is done
    dbg!(v1);
}

In the code above, the .iter() method that is called on the vec, v1 creates an iterator, v1_iter which borrows value immutably. Because the values of v1 were borrowed immutable by the iterator v1_iter it was still possible to use the vec v1 after the iteration by dbg! printing it.

An iterator that borrows the value being iterated mutably

#[test]
fn iter_mut_demo() {
    let mut v1 = vec![1, 2, 3];

    // the .iter_mut() method creates an iterator, 
    // v1_iter which borrows value and can mutate it. 
    let mut v1_iter = v1.iter_mut();

    // access the first item and multiple it by 2
    let item1 = v1_iter.next().unwrap();
    *item1 = *item1 * 2;

    // access the second item and multiple it by 2
    let item2 = v1_iter.next().unwrap();
    *item2 = *item2 * 2;

    // access the third item and multiple it by 2
    let item3 = v1_iter.next().unwrap();
    *item3 = *item3 * 2;

    // end of the iteration
    assert_eq!(v1_iter.next(), None);

    // this will print out [2,4,6]
    dbg!(v1);
}

In the code above, the .iter_mut() method that is called on the vec, v1 creates an iterator, v1_iter which borrows value mutable. That is, the value borrowed can be updated. And the code does exactly that, mutating each value iterated by multiplying it by 2. Because the values of v1 were borrowed mutably and not owned by the iterator v1_iter it was still possible to use the vec v1 by printing it. Printing also confirms that the values in the vec have been updated with each item multiplied by 2.

An iterator that takes ownership of values being iterated

#[test]
fn into_iter_demo() {
    // the .into_iter() method creates an iterator, 
    // v1_iter which takes ownership 
    // of the values being iterated.
    let mut v1_iter = v1.into_iter();

    assert_eq!(v1_iter.next(), Some(1));
    assert_eq!(v1_iter.next(), Some(2));
    assert_eq!(v1_iter.next(), Some(3));
    assert_eq!(v1_iter.next(), None);

    // If the line below is uncommented, the code won't compile anymore
    // this is because, after the iteration, v1 can no longer be used 
    // since the iteration moved ownership
    //dbg!(v1);
}

In the code above, the .into_iter() method creates an iterator, v1_iter which takes ownership, because of this, v1 is no longer usable after the iteration has ownership has moved after the iteration. Uncommenting the dbg! line would lead to a compile error because of this.

Iterator and for…in syntax

So now we have seen that Iterators in Rust can come in 3 forms depending on how ownership is handled when the iterator accesses the values of the data structure being iterated.

How does this then help us answer why the following code:

fn main() {
   let some_ints = vec![1,2,3,4,5];
   for i in some_ints {
       dbg!(i);
   }

   dbg!(some_ints);
}

which looks legit get flagged by the borrow checker?

The answer lies in the fact that the for…in syntax, by default, calls the .into_iter() method, which returns an iterator that takes ownership of values being iterated.

The code above is essentially the same as the following code:

fn main() {
   let some_ints = vec![1,2,3,4,5];
   for i in some_ints.into_iter() {
       dbg!(i);
   }

   dbg!(some_ints);
}

Compiling both will lead to the same error.

So to use the for…in syntax and still be able to use the underlying vec, after iteration, we will have to manually call either the iter() or iter_mut() method instead.

For example, using the iter() method with for..in will look like this:

fn main() {

   let some_ints = vec![1,2,3,4,5];
   for i in some_ints.iter() {
       dbg!(i);
   }

   dbg!(some_ints);

This time around, the code compiles fine. And if the code is executed, the contents of some_ints will get printed fine even after the iteration. Something that was not possible before.

Using the iter_mut() method on the other hand enables us to mutate the values as they are being iterated. For example:

fn main() {

   let mut some_ints = vec![1,2,3,4,5];
   for i in some_ints.iter_mut() {
       *i = *i * 2;
   }

   dbg!(some_ints);

}

The above code compiles fine, and when run, will print [2,3,6,8,10] confirming that it was possible to mutate the values while iterating.

To further explore how all the above works, check IntoIterator and the for ... in Syntax in Rust

Summary

An Iterator is a pattern that allows retrieval of elements from another data structure in sequence. In Rust, it is represented by the Iterator trait.
A data structure implementing the Iterator trait interacts with the borrow checker based on how it handles access to the values in the data structure being iterated.

It is possible for the iterator to borrow the value being iterated immutably. This type of iterator is created by calling iter().
It is possible for the iterator to borrow the value being iterated mutably. This type of iterator is created by calling iter_mut().
It is possible for the iterator to take ownership of the value values being iterated. This type of iterator is created by calling into_iter()

By default, the for..in syntax calls the into_iter() which takes ownership of the value being iterated. This is why after iteration, the underlying variable that holds the original vec becomes unusable. This is because ownership has moved away from it.
To use the for..in syntax and still be able to use the variable that holds the original vec, after iteration, then either call iter() or iter_mut()

Introduction to Digital Signature for the Working Developer

noreply@blogger.com (dade) — Sun, 12 Jun 2022 14:34:00 +0000

In this post, we would be looking at digital signatures. It will be a quick, straight-to-the-point overview of some of the essential things a developer should know about digital signatures. How to think about them and what problems they solve. It is targeted at the working developer who needs to be familiar enough with digital signatures to use them, but who does not need to know the gory details of how they are implemented, or how they work internally.

This post contains the following sections:

What is a digital signature
Components of Digital Signature Schemes
Digital Signature Algorithms
Last note: Digital Signatures versus Message Authenticated Codes.

What is a digital signature

Signatures in real life are used to ascertain that a piece of document originated and is authorized by the signer.

Similarly, in the digital world, digital signatures are cryptographic primitives used for the same purpose. That is, to ascertain that some digital data came from, and has been authorized by an entity that claims to have originated it.

The digital data could be in the form of an email, a PDF document, blockchain transactions, etc. the point of digital signature remains the same: ascertain that some digital data is legit by ascertaining it originated from the right origin. This function is called origin authentication.

Apart from origin authentication, digital signatures can also be used to ascertain that the data signed has not been tampered with. This is called message authentication or Integrity.

It is also worth mentioning that digital signatures are cryptographic primitives that fall under the asymmetric cryptography category because they make use of two key pairs: private and public keys.

Let's now take a look at the components of a digital signature scheme.

Components of Digital Signature Schemes

Digital signature schemes are usually made up of three distinct components or algorithms:

Key generation: This component of the scheme consists of the algorithm that takes care of generating the key pair (private/public) that the signer needs to generate the digital signature.
Signing Algorithm: This component is the actual algorithm that generates the digital signature. It takes the message to be signed and uses the private key generated by the key generation algorithm, to create a digital signature.
Verifying Algorithm: This checks the authenticity of digital signatures. It takes the public key generated via a key generation algorithm, together with the signature, generated by the signing algorithm, and returns either success if the signature is valid or an error if invalid.

Having listed the three main broad components of digital signatures, let's now take a look at some of the popularly known digital signature schemes. For completeness' sake, it would also mention schemes that were once used but are no longer recommended due to security reasons.

Digital Signature Algorithms.

RSA (RSA-PKCS1-v1_5 and RSA-PSS)

RSA (Ron Rivest, Adi Shamir, and Leonard Adleman - last names of the cryptographers who proposed the scheme) was the first digital signatures scheme to be articulated in 1977.

RSA was introduced as a scheme for encryption in Introduction to Asymmetric Encryption for the Working Developer. It is asymmetric cryptography primitive, hence uses private/public key pair. When used for encryption, the public key is used to encrypt a message that is then sent to the owner of the corresponding private key. The private key can then be used to decrypt the message.

When RSA is used for digital signature, the signing algorithm involves taking a hash of the message to be signed, and then encrypting the hash to create a ciphertext that serves as the signature. The message and the signature (ciphertext of the message) are then shared.

The verifying part of the scheme involves two steps. The first step is to use the same hashing algorithm to hash the message received from the signer. The second step is to use the corresponding public key to the private key of the signer to decrypt the received signature (that is the ciphertext the signer published with the message). If the hash value generated by the verifier is the same as the one revealed after using the public key to decrypt the signature, then the signature is valid. If it is not equal, then the signature is invalid.

As mentioned in Introduction to Asymmetric Encryption for the Working Developer RSA is usually used with a padding scheme to make it more secure. The first padding scheme that can be used with RSA to create digital signatures is PKCS1-v1_5. This gives rise to the RSA-PKCS1-v1_5 digital signature scheme. This scheme is no longer recommended due to security vulnerability.

A newer standard RSA-PSS which makes use of PSS encoding algorithm is the preferred scheme to use when using RSA for digital signatures.

Schnorr Signature

Schnorr Signature is another digital signature scheme that was proposed in 1990 by Claus P. Schnorr. It is efficient and secure, with no known vulnerability. Although historically, It is not widely used because it was covered by a patent. The patent has since expired in 2008 so adoption will probably see an uptick. This is observed already has Bitcoin recently added Schnorr signatures as part of the digital signature it supports.

DSA and ECDSA

In 1991, the National Institute of Standard and Technology NIST proposed the Digital Signature Algorithm (DSA). For all intent and purposes, DSA is a variant of the Schnoor signature. The sole reason why it was proposed, was to avoid Shnoor's patent restriction. It has no known vulnerability and enjoyed great adoption, although it has since been replaced with the elliptic curve implementation of the scheme, which is known as the Elliptic Curve Digital Signature Algorithm (ECDSA)

EdDSA

EdDSA, which stands for Edwards-curve Digital Signature Algorithm, was proposed in 2008 by Daniel J. Bernstein. It takes inspiration from Schnoor's signature it is also an elliptic curve-based scheme since it makes use of the Edward curve

It is worth mentioning that these schemes mentioned above are by far not the only digital signature schemes in existence. Although, arguably, they are the most common. There exist other schemes like ElGamal signature scheme, the Rabin signature algorithm, or NTRUsign just to mention a few.

Code Examples using RSA-PSS

The code snippets below show how to use RSA-PSS using Deno's implementation of Web Crypto API. Three methods will be used: generateKey, sign, and verify. Which corresponds to the three components of digital signature schemes.

// Key Generation
const keyParams = {
    name: "RSA-PSS",
    // The length in bits of the RSA modulus
    modulusLength: 2048,
    // The public component e. Recommended value is 65537
    publicExponent: new Uint8Array([1, 0, 1]),
    // hash function required by PSS
    hash: "SHA-256",
};

const keyPair = await crypto.subtle.generateKey(keyParams, true, ["sign", "verify"])

// Signing

const message = new TextEncoder().encode("increase interest rate by 2%")

const signParams = {
    name: "RSA-PSS",
    // A long integer representing the length of the random salt to use, in bytes
    // Value can be either 0 or the length of the output of the key's digest algorithm. 
    // For example, if you use SHA-256 as the digest algorithm, this could be 32
    saltLength: 32
}
const signature = await crypto.subtle.sign(signParams, keyPair.privateKey, message)


// Verification

const verifyParams = {
    name: "RSA-PSS",
    // A long integer representing the length of the random salt to use, in bytes
    // Value can be either 0 or the length of the output of the key's digest algorithm. 
    // For example, if you use SHA-256 as the digest algorithm, this could be 32
    saltLength: 32
}

const verificationResult = await crypto.subtle.verify(verifyParams, keyPair.publicKey, signature, message)

console.log(`Signature verification is: ${verificationResult}`);


// Confirm if message is different verification fails

const differentMessage = new TextEncoder().encode("increase interest rate by 20%")

const failedVerification = await crypto.subtle.verify(verifyParams, keyPair.publicKey, signature, differentMessage)

console.log(`Signature verification for different message is: ${failedVerification}`);

Having the above code snippet in index.js file and running via the following command deno run index.js would print out the following:

Signature verification is: true
Signature verification for different message is: false

Last note: Digital Signatures versus Message Authenticated Codes

In Hash Function in Action: Message Authentication Codes I introduced Message Authenticated Codes (MAC). Which involves two parties making use of a shared secret key to ascertain the integrity of messages.

It is worth mentioning that digital signatures are the asymmetric version of message authentication code, with the extra convenience that it does not require parties to perform any key exchange.

Introduction to Asymmetric Encryption for the Working Developer

noreply@blogger.com (dade) — Mon, 06 Jun 2022 11:10:00 +0000

In this post, we are going to look at asymmetric encryption. As mentioned in the previous post on symmetric encryption, encryption is the cryptographic primitive that guarantees confidentiality, by which we mean the ability for two or more authorized parties to communicate, without an unauthorized person being able to decipher the messages being communicated.

In symmetric encryption, both parties need to use the same key for encryption and decryption. This leads to the problem of how to securely get the communicating parties to use the same key. A solution to this is the Key exchange protocol that was discussed in Introduction to Key Exchange for the Working Developer

Asymmetric encryption is different. Unlike symmetric encryption, it does not have the requirement that the same key needs to be used for encryption and decryption. Hence why it is called asymmetric. It makes use of key pairs instead. One is called the private key, the other called the public key.

In this post, we will be looking at how to use asymmetric encryption and what other general information to be aware of. As always it is targeted at the working developer who needs a grounded understanding of these cryptographic primitives without necessarily covering the internals of these primitives.

Asymmetric Cryptography and Asymmetric Encryption

Oftentimes when private/public keys are mentioned, it is often done so in the context of encryption, that is, asymmetric encryption. The reality is that the use of public and private keys is not restricted to encryption and decryption alone, there are other primitives in cryptography that make use of private/public keys.

For example, the Diffie-Hellman Key Exchange, which I wrote about in Introduction to Key Exchange for the Working Developer is an example of a cryptographic primitive that makes use of asymmetric keys but has nothing to do with encryption.

Asymmetric cryptography is the broad term that refers to all the various cryptosystems that do not depend on having two identical secret keys to function. Asymmetric encryption falls under asymmetric cryptography but so do other primitives. I believe this tidbit needs to be highlighted to prevent the assumption that when public and private keys are used, they are always used in encryption and decryption.

With that clarification out of the way, let us dig into asymmetric encryption by taking a broad look at the different asymmetric encryption algorithms.

Asymmetric Encryption Algorithms

Asymmetric encryption makes use of key pairs. A private key and a public key. This pair of keys can be used to achieve confidentiality. This is because messages encrypted by a key in the pair can be decrypted by the other key.

This is how it usually works. The public key, as the name implies, is made public. This can be shared at the beginning of confidential message exchange, or it can be made available free via any other means: email, on a website, etc. The private portion, on the other hand, is kept secret. To communicate confidentially, the public key is used to encrypt a message, which is then sent to the entity that controls the corresponding private key. This encrypted message can only be decrypted by the owner of the corresponding private key.

A two-way encrypted communication then involves the communicating parties using each other's public key to encrypt messages. That is Alice uses Bob's public key to encrypt messages she wants to send to Bob. Bob can then decrypt these messages from Alice. To reply to Alice's confidentiality, Bob also makes use of Alice's public key to encrypt replies that are sent to Alice.

There are a couple of algorithms that implement the ability to encrypt and decrypt using private/public key pairs. For example ElGamal, Paillier cryptosystem, and Cramer–Shoup cryptosystem but by far, the most popular and most used algorithm is RSA cryptosystem.

The acronym "RSA" is from the last names of the three cryptographers Ron Rivest, Adi Shamir, and Leonard Adleman, who proposed the algorithm in 1977.

Like with most asymmetric cryptosystems, the underpinning of RSA and its security is based on mathematics. But since the aim of this post is to provide the necessary information needed for a working developer to form a working mental model of asymmetric encryption, we will avoid a rigorous take but gloss over the mathematical details instead. Readers who are interested in digging more into mathematics can check the additional section of this post. There, I share links that cover some of the mathematics that is fundamental to RSA.

For now, we take a bird's eye view of the RSA algorithm.

How RSA Works

RSA is based on factorization problem. Factorization is the mathematical operation of splitting an integer into products of smaller integers. For example, the numbers 21, 7, and 3 are its factors.

The factorization problem is the observation that it is easy to find an integer given its factors, but the reverse, that is finding the factors of a given number is hard. A small number like 21 in the example above does not demonstrate this difficulty, but given a substantial large integer, finding its individual factors is extremely hard, with no algorithm to speed up the process, and finding such factors essentially involves brute force.

"Can the reader say what two numbers multiplied together will produce the number 8,616,460,799? I think it unlikely that anyone but myself will ever know."

The above challenge was posed by William Stanley Jevons in his 1874 book, Principles of Science. The number becomes known as Jevon's number. It is essentially a factorization problem challenge. It took 15 years to crack. It was factored by Charles J. Busk in 1889 who found that 89,681 and 96,079 are factors of 8,616,460,799.

Factoring is the underlying, presumably hard problem upon which the RSA algorithm is based. An in-depth explanation is beyond the scope of this post, but the general algorithm is outlined below.

Note that the outline below is often referred to as textbook RSA and this is not how RSA is implemented in practice because it is insecure. There are slight improvements mainly in the area of padding that fortifies the algorithm.

The general outline of the algorithm is as follows:

Choose very two large prime numbers. Let's call them p and q
Let N = p * q N is usually referred to as the modulus
Find the Euler totient of p and q, that is T = (p-1)(q-1)
Choose another number e where 1 < e < T and e do not share any factors with N and T
e is used for the encryption process.
Choose another number d where e * d mod T = 1
d is used for the decryption process.
The pair N and e is the public key. This is published
The pair N and d is the private key. This is kept a secret.
To encrypt, a message, represent the message as a numeric value, say m
Then calculate m ^ e mod N = ciphertext c
To decrypt ciphertext c back to m calculate c ^ d mod N

That is the general outline. To see this in practice, let's try it out with 2 prime numbers that are small enough to make demonstration possible. We choose p = 2 and q = 7

Choose very two large prime numbers. Let's call them p and q
- p = 2 and q = 7
Let N = p * q N is usually referred to as the modulus
- N = 2 * 7 = 14
Find the Euler coefficient of p and q, that is T = (p-1)(q-1)
- T = (2-1)(7-1) = 1 * 6 = 6
Choose another number e where 1 < e < T and e do not share any factors with N and T
- e = 5
e is used for the encryption process.
Choose another number d where e * d mod T = 1
- d = 11 because:
- 5 * 11 mod 6 = 55 mod 6 = 1
d is used for the decryption process.
The pair N and e is the public key. This is published
- Public key is (5, 14)
The pair N and d is the private key. This is kept a secret.
- Private key is (11, 14)
To encrypt, a message, represent the message as a numeric value, say m
- Let the message be B which in numeric form we have as 2
Then calculate m ^ e mod N = ciphertext c
- 2 ^ 5 mod 14 = 4
- Cipher text is 4
- Which in our encoding scheme is D
To decrypt ciphertext c back to m calculate c ^ d mod N
- 4^ 11 mod 14 = 2
- Plain text is 2
- Which in our encoding is B - which is the original value encrypted

The underlying security of this algorithm is the fact that given the value N which is public, it is practically impossible to figure out the factors p and q. If it were possible then it would be possible to derive d which would make the whole scheme insecure.

As mentioned earlier, textbook RSA is not used in practice as it is not secure. For example, using textbook RSA to encrypt small messages is insecure, because an eavesdropper can encrypt all the small numbers and quickly observe if any of their encrypted numbers match the ciphertext. This is why in practice, RSA makes use of padding schemes that pad the message before encryption.

The very first such a padding scheme in PKCS#1. This padding scheme has been shown to also be insecure and should not be used. Hence in practice, the padding scheme recommended to be used with RSA is the Optimal Asymmetric Encryption Padding (OAEP). Hence RSA-OAEP

Let's now see how RSA-OAEP looks with a working code example.

Example of RSA-OAEP using web crypto API

In the code sample below, we generate a private/public key pair, use it to encrypt a text, and then decrypt it to retrieve the original plaintext. The code makes use of web crypto API so it should work in most browsers, Deno, or the latest version of node.

let plainText = "Hello world!";

const keys = await window.crypto.subtle.generateKey({
   name: "RSA-OAEP",
   // The length in bits of the RSA modulus
   modulusLength: 2048,
   // The public component e. Recommended value is 65537
   publicExponent: new Uint8Array([1, 0, 1]),
   // hash function required by OAEP
   hash: "SHA-256",
},
   // A boolean value indicating whether it will be possible
   // to export the key using SubtleCrypto.exportKey() or
   // SubtleCrypto.wrapKey().
   true,
   // An Array indicating what can be done
   // with the newly generated key
   ["encrypt", "decrypt"]
)

// We encrypt
const ciphertext = await window.crypto.subtle.encrypt(
   { name: "RSA-OAEP" },
   keys.publicKey,
   new TextEncoder().encode(plainText)
)


// We convert ciphertext to a hex string and console logged it
console.log(buf2hex(ciphertext))

// We decrypt
const decrypted = await window.crypto.subtle.decrypt(
   {
       name: "RSA-OAEP"
   },
   keys.privateKey,
   ciphertext
)

// We convert to string
const decryptedPlaintext = new TextDecoder().decode(decrypted);
console.log(`Encrypting and decrypting was successful: ${plainText === decryptedPlaintext}`)

// utility to convert ArrayBuffer to hex string
function buf2hex(buffer: ArrayBuffer) {
   return [...new Uint8Array(buffer)]
       .map(x => x.toString(16).padStart(2, '0'))
       .join('');
}

Executing the following code, for example using Deno should give the following output:

deno run index.ts
Check file:///Users/dadepoaderemi/Documents/delete/blogging/assymmetric/index.ts
3b69df6eee76c660b8c9c307f89c4a023f455b06cf8da6622c9ff2073...
Encrypting and decrypting was successful: true

Drawbacks of RSA and Introduction to Hybrid Encryption

RSA is inefficient. It can only encrypt a small amount of plaintext at a time, and the speed of encryption and decryption is quite slow. This drawback is not limited to RSA but applies to most asymmetric encryption algorithms. This is mainly because these algorithms implement math operations.

Because of these drawbacks, asymmetric encryption is usually not used to encrypt messages. Instead, it is used as part of what is usually referred to as hybrid encryption constructions.

The general idea behind hybrid encryption is this: use symmetric encryption for the actual encryption since symmetric encryption is more efficient while asymmetric cryptography is used to exchange the symmetric keys. The part where asymmetric cryptography is used to ensure the parties communicating have the same key is usually called key encapsulation, while the actual encryption via symmetric encryption is called data encapsulation.

Asymmetric encryption is an example of asymmetric cryptography that can be used for key encapsulation. In this case, if Alice wants to communicate confidentially with Bob, she generates a symmetric key and then encrypts it using Bob's public key. This ensures Bob can decrypt with his private key and retrieve the symmetric key to be used for encryption.

Another scheme is to use Diffie Hellman key exchange, as the asymmetric cryptography system used in exchanging the symmetric key used for actual encryption and decryption.

The idea of hybrid encryption is very practical, this has led to the creation of well-defined schemes that makes use of hybrid encryption but only expose the typical asymmetric encryption interface to the use. This means the user of such schemes only need to worry about the private and public keys, while internally the scheme makes use of symmetric encryption. This also means the user does not have to manually perform multiple steps that consist of key encapsulation and data encapsulation but only uses the private and public keys to encrypt and decrypt.

There are many hybrid encryption schemes in existence, although arguably the most widely adopted standard is Elliptic Curve Integrated Encryption Scheme (ECIES). Hence I will quickly show how ECIES looks in practice.

Unfortunately, Web Crypto API does not support ECIES, so for this demonstration, I will use nodejs and the eciesjs package.

Create a new node js project and install eciesjs by running the following command

npm init ecies_demo -y
npm install eciesjs

Then in the index.js have the following code:

import { encrypt, decrypt, PrivateKey } from 'eciesjs'

const prvKey = new PrivateKey()
const pubKey = prvKey.publicKey
const message = Buffer.from('message to b')

// encrypt the message
const encryptedToB = encrypt(pubKey.toHex(), message);

// console log the encrypted message as hex
console.log(encryptedToB.toString('hex'))

// decrypt the message
const decrypted = decrypt(prvKey.toHex(), encryptedToB).toString()

// confirm original message and decrypted message is same
const isSame = message.toString() === decrypted
console.log(`decrypted and original message is same: ${isSame}`)

This code uses a generated public key to encrypt a message which is then decrypted with the corresponding private key and confirmed that the original message and decrypted message are the same.

As can be seen, the interface only exposes the usage of public-private key pairs and hides the fact that symmetric encryption is being used. In fact, the ECIES scheme is more complex and makes use of other cryptographic primitives like key-derivation function (KDF) and HMAC,

A functional diagram of ECIES can be seen below:

Source https://core.ac.uk/download/pdf/36042967.pdf

The good thing is that the end-user does not need to deal directly with this complexity. A simpler interface based on private and public keys is exposed. This is the advantage of using a hybrid encryption scheme like ECIES.

Additional Resources

Here are some of the link that covers some of the fundamental mathematics that makes asynchronous encryption work

A Neophyte's Introduction to the async/await landscape in Rust

noreply@blogger.com (dade) — Sun, 29 May 2022 20:30:00 +0000

The truth of the matter is, the async/await features in other programming languages seem to be more straightforward than it is in Rust. For example in JavaScript, you have the syntax that you use, and that is about it. But in Rust, there seem to be more moving parts.

Tokio? Future? There is a futures crate? Runtimes? How do all these fit into the async/await picture in Rust?

This is a short post to help clarify some of the terms a developer would encounter when they start exploring asynchronous programming in Rust. It would explain the main moving parts, which are: the standard library's std::future::Future, the futures crates, and asynchronous runtimes like tokio and async-std.

Let get started:

std::future::Future

The crux of asynchronous programming is to maximize computing resources, especially IO-bound computation. One of the ways this is done, is to code in such a way that the computation is not directly executed but written in a suspended form that can be later executed or paused, or retried.

The type used to express such a computation is std::future::Future. For example, a print statement that is not executed directly can be written is as follows:

use std::future::Future;
fn future_print() -> impl Future<Output = ()> {
    async {
        println!("Hello World!");
    }
}

The return type impl Future<Output = ()> means this is a function that returns any type that implements the std::future::Future trait.

To fully understand the syntax which might be a bit obscure, check the Trait section of the Rust book.

The std::future::Future is part of the standard library and there is no need to depend on any external crate to have access to it.

To confirm that the future_print function does not execute directly, let us try and call it in main to see what happens:

use std::future::Future;

fn future_print() -> impl Future<Output = ()> {
    async {
        println!("Hello World!");
    }
}

fn main() {
    future_print();
}

The output should be something similar to:

warning: `async_await` (bin "async_await") generated 2 warnings
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/async_await`

Process finished with exit code 0

As you can see, there is no Hello World printed to the console. Confirming that the std::future::Future denotes computation that has been described but that is not executed directly and this is the crux of asynchronous programming in Rust.

In practice, there is usually no need to use the std::future::Future type directly as the return type. Rust has a syntactic sugar in the form of the async keyword that makes the code less noisy. The above code snippet can be re-written as:

async fn future_print() {
    println!("Hello World!");
}

fn main() {
    future_print();
}

This is less noisy, even though it is essentially the same as the verbose code that directly uses the std::future::Future type.

The question now is, how do a method that returns a future be executed? To answer this we look at the next piece of the async/await story in Rust. The futures crate, which can be found here

futures crate.

Formally futures-rs. This crate provides the other core foundational components needed for asynchronous programming in Rust.

The components it provides, are not included in the standard library like std::future::Future, and hence to bring them in, the crate needs to be defined as a dependency in Cargo.toml. It can be done as follows:

[dependencies]
futures = "0.3.21" # change to the current version

Even though it is not included in the standard library it is worth noting that the futures-rs crate is developed by the Rust core team, as can be seen from the fact that the code repository is under the rust-lang namespace on Github

The futures-rs crates include key trait definitions like Stream, as well as utilities like join!, select!, and various futures combinator methods that enable expressive asynchronous control flow.

This post won't look at the utilities like join!, select!, instead we go back to the initial question we are yet to answer: We have a function that returns a Future how do we run it?

To run a Future in Rust, there is a need for an asynchronous runtime. This runtime is usually called the executor. The executor is a machinery that is tasked with the management of all the resources required for executing futures. It takes a computation described as a Future and ensures it is executed asynchronously.

The futures-rs crate comes with such an executor which can be accessed via futures::executor::block_on.

The following code snippet shows its usage:

use futures::executor::block_on;

async fn future_print()  {
    println!("Hello world");
}

fn main() {
    block_on(future_print());
}

Now if the following code is executed, the output should look like the following:

/Users/dadepoaderemi/.cargo/bin/cargo run --color=always --package async_await --bin async_await
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/async_await`
Hello world

Process finished with exit code 0

Confirming that we can execute the future code with the usage of futures::executor::block_on

At this point, we have looked at std::future::Future which is part of the standard library and is used to describe computation to be executed later by a runtime. We also looked at the futures-rs crate, which provides such a runtime in the form of the futures::executor::block_on executor.

The only problem is that the futures::executor::block_on executor is quite primitive and not powerful enough to be used in a real life production environment.

To fill the gap for more powerful, feature-rich asynchronous runtimes, various crates have spurned up within the Rust ecosystem. Some popular ones include: Tokio, async-std, and smol.

They all do the same thing: Provide a sophisticated and more powerful runtime that executes asynchronous computations.

We will use Tokio to demonstrate. Update the dependencies as follows:

[dependencies]
futures = "0.3.21" # change to the current version
tokio = { version="1.18.2", features = ["full"] } # change to the current version

Note that using external runtimes like tokio does not necessarily remove the need for the futures-rs crate, as it contains useful utilities that can still be used alongside the runtime.

Using the tokio runtime would look something like this:

use tokio::runtime::Runtime;

async fn future_print() {
    println!("Hello world")
}

fn main() {
    Runtime::new().unwrap().block_on(future_print());
}

This works. The only problem though is that this is a bit verbose. To make the code less noisy, we use another Rust language keyword related to async/await, the .await, and a macro from Tokio: #[tokio::main].

In code, it looks like this:

async fn future_print() {
    println!("Hello world")
}

#[tokio::main]
async fn main() {
    future_print().await;
}

Running the above code would print something like this into the console:

/Users/dadepoaderemi/.cargo/bin/cargo run --color=always --package async_await --bin async_await
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
     Running `target/debug/async_await`
Hello World!

Seeing the Hello World! confirms that we were able to use a runtime provided by Tokio runtime to execute the computation.

Summary

Using async/await in Rust is not as straightforward as just using the language syntax as it is with languages like JavaScript or C# etc that have similar asynchronous constructs.

More moving parts might not be obvious to the beginner Rust developer. This post gave a quick overview of the crucial components:

The std::future::Future type is part of the standard library that forms the crux of the asynchronous programming model as it is used to denote computation that can be executed asynchronously.

The futures-rs crates, maintained by the Rust team, provide more of the foundations for asynchronous programming in Rust. It comes with a simple executor that can be used as a runtime to execute asynchronous code.

And finally, we looked at Tokio, as an example of an external crate that provides a more sophisticated asynchronous runtime that is usually used in a production-like environment.

Other well known crates providing asynchronous runtimes include smol, and async-std.

In summary, the core async/await landscape can be described as comprising the standard library type of std::future::Future which denotes asynchronous computation. The futures-rs crate provides additional foundational components: traits, macros utilities, etc and then finally the 3rd party runtimes that are external to the standard library which provide sophisticated executors used for running asynchronous code.

This post does not attempt to go into details about how to use async/await in Rust. It seeks to just be the introductory text that explains the lay of the land. To learn more about asynchronous programming in Rust, you can check the following resources:

Introduction to Key Exchange for the Working Developer

noreply@blogger.com (dade) — Sat, 07 May 2022 19:37:00 +0000

In Introduction to Symmetric Encryption for the Working Developer, we saw how to encrypt and decrypt data. Being symmetric encryption, both the process of encryption and decryption needed the same key. This poses a challenge, which was not addressed in the post: How do one get across the same key to both parties who want to engage in encrypted communication?

Basically how do we perform key exchange? How do we get the same key to parties who want to encrypt and decrypt messages?

An approach is perhaps for the parties involved to first meet in person and exchange the key?

Another approach could be to use another communication channel to first share the key? But then again, if the other communication channel is digital, and it uses symmetric encryption, how should its encryption key be also exchanged? This becomes a catch 22 real quick.

This is the problem Key exchange schemes seek to solve, and thankfully there is a secure way to have two parties exchange secret keys without the need to physically meet in person. This is what this post is about. We will be looking at two popular mechanisms for key exchange: Diffie-Hellman key exchange procedures and application of RSA for key exchange.

As always, as with the other posts in this series, the idea is to provide the basic information needed by the working developer to be able to understand and use these cryptographic primitives without going into the thick of their internal details or implementation.

This post contains the following sections

Entering the realm of Public Key Cryptography
Introduction to Diffie-Hellman Key Exchange: An Intuition.
Whirlwind tour of the Mathematics
Diffie-Hellman in code
Diffie-Hellman Standards
Using RSA for key exchange
Conclusion and References

Entering the Realm of Public Key Cryptography

Public key cryptography (or Asymmetric key cryptography) is a class of cryptography primitives that makes use of dissimilar keys. This is different from Symmetric key cryptography which makes use of similar keys for its operations.

Public key cryptography also relies more heavily on Mathematics, especially number theory, and the security properties and guarantees it provides is often as a result of a direct consequence of some mathematical theory. Hence in discussion of cryptographic primitives that can be classified as public key cryptography there is more encounter of mathematical notations and theorems.

This post will not go into the mathematical underpinning of the cryptography primitives discussed. Instead, I will highlight the mathematical theories in this post and point to resources, motivated readers can check out to learn more.

Let's start by exploring one of the two schemes that uses public key cryptography for key exchange: that is: Diffie-Hellman.

Introduction to Diffie-Hellman Key Exchange: An Intuition

The Diffie-Hellman key exchange is a method that allows two parties, communicating in a public, insecure channel, to exchange messages that allows them to derive the same secret key. But the method has the property that eavesdroppers observing the messages being exchanged won't be able to use it to construct the secret key derived by the parties engaged in the key exchange.

How is this possible? To have two parties exchange two numbers, that anyone can see, and then, separately, these two parties use the numbers they receive from each other to compute another number privately, and the number they compute will end up being the same (ie the secret key) but the people that observed the numbers that was exchange cannot do the same computation to arrive at the secret key?

A good intuition to have about how the process works is this. It is a process for two parties to first exchange two secret numbers. But they do not just exchange this secret number, they first mix it up with another number both parties agree with.

So each party comes up with a secret number, mixes it up with a publicly agreed number and shares.

After the sharing, both parties have the secret number generated by the other party mixed up with an agreed number. Anyone observing the exchange can only see the results of the mixed number (that is, party A's secret number mixed with an agreed number which is not secret and party B's secret number mixed with the same agreed number). The security of this exchange boils down to the inability to break apart the result of this number mixing to reveal the secret number that was mixed with it. In Diffie-Hellman, this is impossible to achieve based on the discrete algorithm problem which will be briefly explored later on. But for this intuition, we agree that, parties mixed numbers and share, and anyone observing cannot break this number apart.

After the mixing and sharing what do we have? We actually have both parties having each other's secret number. Only that, this secret number they have of each other is not "bare" but mixed up with an agreed number.

What then happens is that both parties perform a computation using their secret number on the other party's secret number (but mixed up in an agreed number) and they will both arrive at another secret number which can be used as the secret key.

That is the intuition. The Diffie-Hellman is a process for two parties to first initially exchange a secret number mixed up with an agreed and non-secretive number, and then use this to perform a computation. What ends up happening is that the inputs to the computation on each party's side are the same. That is for party A, it is compute(secret A, M1 = (mixing number + secret B)) while for party B, it is compute(secret B, M2 = (mixing number + secret A)).

Even if the mixing number is known, as long as it is impossible to break M1 or M2 apart to retrieve secret B or secret A, then the scheme works.

That is the intuition of the process. Now let's dig deeper into how things work, without also going into too much of the mathematics.

Whirlwind tour of the Mathematics

The idea of this section is to give an overview of the key concepts from mathematics that come together to make Diffie-Hellman possible. For obvious reasons, an in depth coverage of these concepts is not possible in this one post, but links would be sprinkled in pointing to where more information can be gotten.

So let's start.

We all know about sets from elementary mathematics. When sets fulfil certain characteristics, they are said to form a group. These characteristics include Closure, Associativity, Presence of an Identity Element and presence of an Inverse element. The study of these characteristics is called group theory.

A key part of a group is the presence of a binary operation. That is an operation that operates on two inputs to produce an output that is also a member of the set. Examples of binary operations include addition, or multiplication or Modulo multiplication. Modulo operation are mathematical operations involving numbers that wrap around. It is also often called clock arithmetics.

The two flavors of Diffie-Hellman algorithms that we have, Diffie-Hellman in a modulo multiplicative group and Elliptic Curve Diffie Hellman (ECDH) both make use of groups and binary operations.

In the first flavor, the group includes the set of strictly positive integers 1, 2, p – 1 for p a prime number, along with modular multiplication, where the modulo is a prime.

The second flavor makes use of groups that form an elliptic curve, while the binary operation is addition. The algorithm for ECDH is a bit involved when compared hence a quick overview of the first flavor is given in this post.

So as stated, the first (or classic) Diffie-Hellman algorithm makes use of groups made up of positive integers, with binary operation being multiplication with the modulo being a prime P.

The algorithm involves picking a number that forms a primitive root with the modulo prime. This number that we pick when multiplied with itself ad-infinitum, in modulo prime, will generate a cyclic subgroup, ie when it is raised to power n, where n = 1,2,3, (infinity - 1), under modulo prime P, will produce a cyclic group. This extra characteristic is important in order to ensure the discrete logarithm problem is sufficiently hard enough to ensure the security of the algorithm.

And the discrete logarithm problem is a class of problems that is sometimes referred to as one way function, i.e. functions easy to compute given inputs, but hard or almost impossible to retrieve the inputs given the results of the computation. The discrete logarithm problem simply states that, given a number x and y where y is greater than x, it is hard to find the number of times n, x needs to be multiplied by itself to result into y, under modulo of another number p.

In the classic Diffie-Hellman algorithm, a prime number P and an integer g, the generator is agreed upon by the two parties. This is what they use to mix the secret number they want to pass on to each other that would be used to derive the private key.

Party A generates a secret number sa, and then mixes it up with prime p and generator g by computing = AK = g^sa mod p

AK is referred to as a public key and Party A sends this value to party B.

Party B generates a secret number sb, and then mixes it up with prime p and generator g by computing = BK = g^sb mod p

BK is referred to as a public key and party B sends this value to party A.

At this point, party A has BK, while party B has AK.

Party A now computes the secret key by computing SK= BK^sa mod p and party B also computes the secret key by computing SK= AK^sb mod p. They both arrive at the same value because:

BK^sa = (g^sb)sa = g^(sb)(sa) = (g^sa)sb = AK^sa mod p

The two values publicly transmitted are AK = g^sa mod p and BK = g^sb mod p.

g and p are known, AK and BK are also known, as they are communicated in the clear. But anyone observing these numbers won't be able to derive what sa or sb is. Because of the discrete logarithm problem.

It is worth mentioning that in practice, the SK arrived is not used directly as the secret key, but usually passed through a Key derivation function.

Let's see how this procedure looks in code.

Diffie-Hellman in code

Before showing the code, let's go over the steps again.

As stated, the objective is for two parties to arrive at the same secret key that can be used for symmetric encryption, without having to directly share the key.

A rough step by step outline of how it works is as follows:

Both Alice and Bob agree on using Diffie-Hellman. This involves agreeing on two numbers. One small usually referred to as g, and another big, which is a prime number, usually referred to as p.
Alice generates a secret number. Performs a modulus computation with it together with g and the prime number p. This computation derives another number called the public key. This is why this procedure falls under Public Key Cryptography.
Bob also does the same.
Alice gives Bob her public number, while keeping her secret one. Bob also gives Alice his public number and keeps his secret number. Now Alice has her secret number and technically Bob's secret number mixed within Bob's public number. Bob also has his secret number and Alice's secret number mixed in the public number he received from her.
Alice uses her secret number and Bob's public number to calculate another number
Bob does the same, using his own secret number and Alice's public number.
Both Alice and Bob will then arrive at the same generated number which can then be used for symmetric encryption.
Eve who observed all the exchanges, could know the algorithm being used and can see all the numbers exchanged, including the g and p agreed upon, won't be able to compute the shared secret both Alice and Bob compute independently.

The following Javascript code snippets in nodejs demonstrates how the Diffie-Hellman scheme can be put to use.

const crypto = require('crypto')
const assert = require('assert');


// -- Select mod p group

// Alice is using the modp15 group
const groupA = crypto.getDiffieHellman('modp15')
// const groupA = crypto.createECDH('brainpoolP160t1')

// Bob is also using the modp15 group
const groupB = crypto.getDiffieHellman('modp15')
// const groupB = crypto.createECDH('brainpoolP160t1')

// The modp15 group just like others define same generator and prime. hence the prime and generator for groupA and groupB should be same
assert(groupA.getPrime().toString("hex") === groupB.getPrime().toString("hex"))
assert(groupA.getGenerator().toString("hex") === groupB.getGenerator().toString("hex"))


// -- Generate the keys

// With this call, the secret key k1 is generated
// The k1 is then use to generate the public key, which is = g^k1 mod p
// Note that new k1 is generated on each invocation of this function.
groupA.generateKeys()

// With this call, the secret key k2 is generated
// The k2 is then use to generate the public key, which is = g^k2 mod p
// Note that new k2 is generated on each invocation of this function.
groupB.generateKeys()


// At this point the public key can be shared with other parties which can then be used to arrive at the shared secret
// Not that the private key is also available via getPrivateKey call but there should be no need to retrieve this. Having
// this value known by a malicious actor will compromise the integrity of the protocol
console.log(`Public key of group A ${groupA.getPublicKey().toString("hex")}`)
console.log(`Public key of group B ${groupB.getPublicKey().toString("hex")}`)


// --- Now each party use the other public key to generate the shared secret

let sharedSecretA = groupA.computeSecret(groupB.getPublicKey())
let sharedSecretB = groupB.computeSecret(groupA.getPublicKey())


// Confirm that same value was arrived at

assert(sharedSecretA.toString("hex") === sharedSecretB.toString("hex"))

Diffie-Hellman Standards

The Diffie-Hellman scheme is made up of components that can be varied. Depending on how these components are configured, the security guarantees of the scheme can be affected.

The main components that are configurable is the value of g (the generator), and p (the prime) to be used.

Thing is, anyone implementing Diffie-Hellman key exchange is free to pick whatever value they think is secured for these two components. For example this file contains some popular Diffie-Hellman configuration observed in the wild.

The only problem is, not all values lead to a secured scheme. For example in 2016, it was discovered that socat a networking utility, was using Diffie-Hellman configuration that makes it insecure. Hence as a developer, implementing or using any service that makes use of Diffie-Hellman key, to be aware what secured configurations are being used.

To help with this, there are recommendations stated in various RFCs. These recommendations are largely categorized into two: Recommended values to be used for multiplicative groups and those based on elliptic curves.

Recommendation for DH For the multiplicative groups, it is advised to use prime numbers whose values are not less than 2048 bits. This ensures that a minimum of 128 bit of security is provided. The recent RFC that contains up to date best practices and recommendations can be found in RFC 7919

Recommendation for ECDH For the Elliptic curve diffie-hellman, there are a couple of curves that most applications have standardized around. These include P-256, Curve25519, and Curve448. The curve P-256, and Curve25519 provides 128 bit of security, while Curve448 offers around 224 bit of security. When Curve25519 is used to implement the Diffie-Hellman key exchange, it is also referred to as X25519. ECDH is preferred over DH due to its smaller key size. To get a 128 bit of security DH requires nothing less than 2048 bits, but with ECDH, the requirement is nothing less than 256 bits.

Using RSA for key exchange

RSA is a popular asymmetric crypto-system which can also be used for exchanging keys. RSA by itself is a distinct topic that should be covered separately but for completeness it is mentioned here as another way for two parties to exchange secret keys.

Conclusion and References

Summary

Two parties can arrive at the same secret number by performing the Diffie Hellman key exchange. This helps solve the problem of how two parties securely share the secret key needed for encryption.
Diffie Hellman requires numbers that form what is mathematically known as groups. Two popular groups are widely used: Multiplicative groups modulo a prime number and Elliptic curves
When Multiplicative groups modulo a prime number is used, it is usually referred to as just Diffie-Hellman ie DH
When Elliptic curves are used, it is usually referred to as Elliptic Curve Diffie Hellman ie ECDH
The recommended standard to use for DH is found in RFC 7919
The popular curves used in ECDH are P-256, Curve25519, and Curve448, with Curve25519 being quite popular.
When Curve25519 is used to implement the Diffie-Hellman key exchange, it is also referred to as X25519.
ECDH is preferred over DH due to its smaller key size. To get a 128 bit of security DH requires nothing less than 2048 bits, but with ECDH, the requirement is nothing less than 256 bits.

Mathematical Concepts

Introduction to Authenticated Encryption for the Working Developer

noreply@blogger.com (dade) — Thu, 05 May 2022 16:06:00 +0000

In Introduction to Symmetric Encryption for the Working Developer, I presented an overview of symmetric encryption. Which as explained in the post, is a process for establishing confidentiality during communication between two parties.

The only problem is that just encrypting data using symmetric encryption does not provide all the security requirements needed during secure communication.

The issue is that, it is still possible for an eavesdropper to tamper with the message even without being able to make sense of the ciphertexts, and the receiving party won't be made aware that the message they are decrypting has been tampered with and is not exactly the original one that was sent.

Hence even though encryption gives us confidentiality, done alone, it does not guarantee the other security requirements like authenticity/integrity that is often needed - i.e. it does not guarantee that a message came from the right party (authenticity) and it has not been modified in any way (integrity).

Encryption alone is not enough, we need authenticated encryption. This post will introduce the concept of authenticated encryption.

What is Authenticated Encryption?

In Hash Function in Action: Message Authentication Codes a similar argument was made regarding hash functions. Where it was stated that hash functions are never enough and there is the need for message authentication to also ensure things like authenticity and integrity.

How then can authenticated encryption be achieved? Do we use a similar approach that was used in message authentication?

Well, an approach to achieving authenticated encryption is to take the output of encryption and then apply message authentication code to it.

For example, using HMAC to create an authentication tag (hash/digest of the cipher text)). Both the ciphertext and the authentication tag is then sent during communication.

In general such manual constructions are referred to as EtM (Encrypt-then-MAC) and a specific construction of it would be AES-CBC-HMAC. Which is a construction where you encrypt using AES-CBC and then create an authentication tag using HMAC.

The problem with this approach is that it is error prone. It is fickle and possible to make simple errors that lead to tremendous security issues.

For example one needs to ensure that the same key is not used for both encryption and generating the HMAC.

Also special care needs to be taken that the IV values used for the AES are truly random and never reused. Also because two separate cryptographic primitives need to be separately implemented, it does not make for the most efficient approach.

Because of these problems with manually constructed EtM schemes, it is not recommended to manually construct authenticated encryptions. Instead the safe advice is to make use of all-in-one constructions that simplify the process of having authenticated encryption.

This is what is discussed in the next section.

All-in one Authenticated Encryption Constructions and Associated Data

All-in-one authenticated encryptions can either be based on Advanced Encryption Scheme (AES) or using another algorithm apart from AES.

The all-in-one construction for achieving authenticated encryption that builds on AES does so by having an AES mode of operations that include authentication. For more about modes of operation, see Introduction to Symmetric Encryption for the Working Developer.

They do so in such a way as to hide the complexity and the various moving parts, presenting a much simpler interface to developers.

These all-in-one constructs also provide the ability to optionally attach some metadata with the ciphertext, if that is the case, then they are called Authenticated Encryption with Associated Data (AEAD).

An example of such a mode that supports both authentication with the ability to attach associated data is Galois/counter mode (GCM).

The associated data that is attached is unencrypted.

What then is the use you might ask?

The associated data is for scenarios where you have data that does not need to be hidden, but must go along with some other data that needs to be hidden.

An example of such a scenario is a network packet, where the body which contains the data should be encrypted, while the headers, which do not contain any sensitive data and are used by network equipment like routers, can remain in plaintext. But even though the headers are in plaintext, each header needs to be associated with the encrypted data it belongs with. Using AEAD, the header can be the associated data. This means that the same associated data used during encryption, needs to be provided on decryption. If the associated data has been tampered with, the decryption process will fail.

The associated data is a flexible component of AEAD, and can be used for anything. The general idea is that it provides some sort of encryption context. This encryption context can be whatever that is required for it to be.

There are other modes of operation that can be used with AES that are AEAD. This Wikipedia link provides an overview of some of this mode.

There is also ChaCha20-Poly1305 which is an AEAD. It is a separate encryption scheme and not just a mode of operation of AES.

Even though ChaCha20-Poly1305 is newer than AES and not a standard, it has grown to be a solid scheme and a popular alternative to AES.

The rest of this post looks at AES-GCM and ChaCha20-Poly1305.

AES-GCM Authenticated Encryption and Associated Data.

AES-GCM is an AEAD. It is constructed from AES used with the Galois/Counter Mode. By it being an AEAD, it means it provides authenticated encryption out of the box (the AE part in AEAD), with the ability to add associated data (the ED part in AEAD).

The Web Crypto API supports AES-GCM, hence can be used within a browser or Deno. Using it is similar to the code snippet presented in Introduction to Symmetric Encryption for the Working Developer. The only difference is, instead of passing in "AES-CBC" and the algorithm identifier, the string "AES-GCM" is used instead. The updated code is reproduced below. It also makes use of associated data.

// We generate a key
// See https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/generateKey
let key = await crypto.subtle.generateKey({
   name: 'AES-GCM',
   length: 128
}, false, ['encrypt', 'decrypt']);

// We generate random 16 bytes to be used as initialization vector
let iv = new Uint8Array(16);
// and fill it with random numbers
// see https://developer.mozilla.org/en-US/docs/Web/API/Crypto/getRandomValues
await crypto.getRandomValues(iv);

// The web crypto requires data to be in ArrayBuffer
let plaintext = new TextEncoder().encode("hello world");

// The encryption and decryption requires specifying certain parameters:
// the algorithm and initialization vector, associated data etc
// see https://developer.mozilla.org/en-US/docs/Web/API/AesGcmParams
let associatedData = new TextEncoder("utf-8").encode("Greeting");

let param = {
   name: 'AES-GCM',
   iv,
   associatedData,
};

// We encrypt the plaintext to get the ciphertext
let ciphertext = await crypto.subtle.encrypt(param, key, plaintext);

// We decode and console logged the ciphertext
console.log(new TextDecoder("utf-8").decode(ciphertext))

// We decrypt the ciphertext back to plaintext
let deCryptedPlaintext = await window.crypto.subtle.decrypt(param, key, ciphertext);

// We tranform the plaintext from bytes to string and then print it out
console.log(new TextDecoder("utf-8").decode(deCryptedPlaintext));

// Causing decryption to fail by providing different associated data

// Create another associated data and param
let wrongAssociatedData = new TextEncoder("utf-8").encode("Payment");

let wrongParam = {
   name: 'AES-GCM',
   iv,
   wrongAssociatedData,
};


// Decryption would fail because the associated data used for decryption "Payment"
// is different from one used for encryption "Greeting"
await window.crypto.subtle.decrypt(wrongParam, key, ciphertext);

The code above makes use of AES-GCM as the AEAD. It provided an associated data. It demonstrates encryption and decryption. It also shows that the decryption will fail, if a different associated data, from the one used at encryption is attempted for decryption.

ChaCha20-Poly1305

Unlike AES-GCM, (which is an AES), ChaCha20-Poly1305 is an entirely different encryption scheme designed by Daniel J. Bernstein. It is really a combination of a stream cipher (ChaCha20) and a MAC (Poly1305).

Even though ChaCha20-Poly1305 is not standardised by NIST, it has been specified as an RFC by Google in rfc7539. It has also seen massive adoption in the industry, being supported in protocols like OpenSSH, TLS etc. Hence it is a battle tested encryption scheme and can be safely used.

As of this writing, Web Crypto API does not support ChaCha20-Poly1305, so by extension is not available in a browser environment or Deno.

On the other hand, ChaCha20-Poly1305 can be used from Node since the Node crypto API depends on OpenSSL. So if you have the version of OpenSSL that supports ChaCha20-Poly1305, it can be used via Node.

The code snippet below shows how to use ChaCha20-Poly1305 in Node.

const { randomBytes, createDecipheriv, createCipheriv } = await import('node:crypto');

// chacha20-poly1305 requires 256 bit (32 byte) as key 
const key = randomBytes(32);
// Also a 96 bit iv/nonce
const iv = randomBytes(12);;
// create some associated data
let aad = Buffer.from("Greeting")

// create the cipher
let cipher = createCipheriv('chacha20-poly1305', key, iv, { authTagLength: 16 });

// Add the associated data to the cipher
cipher.setAAD(aad, {
  plaintextLength: Buffer.byteLength(plaintext)
})

// Data to encrypt
let plaintext = "Hello world";

// Encrypt and finalize encryption
const ciphertext = cipher.update(plaintext);
cipher.final()

// Console log the cipher text in hex
console.log("ciphertext:", ciphertext.toString('hex'))

// Decrytpion

// Create the decipher using exactly same parameters used to create the cipher
let decipher = createDecipheriv('chacha20-poly1305', key, iv, { authTagLength: 16 });
// Get the auth tag from the cipher and set it on the decipher
// This ensures the authentication done is done on decryption
decipher.setAuthTag(cipher.getAuthTag())
// We set the same associated data used in encryption
decipher.setAAD(aad);
// We start the decryption
let decrypted = decipher.update(ciphertext)
// We finalise the decryption and save the plaintext in a variable
decrypted = Buffer.concat([decrypted, decipher.final()])
// Print out the plaintext
console.log("plaintext:", decrypted.toString())

console.log("Encryption then Decryption Works", plaintext.toString() === decrypted.toString());

// Causing decryption to fail by providing different associated data

// Create the decipher using exactly same parameters used to create the cipher
let wrongDecipher = createDecipheriv('chacha20-poly1305', key, iv, { authTagLength: 16 });
// Get the auth tag from the cipher and set it on the decipher
// This ensures the authentication done is done on decryption
wrongDecipher.setAuthTag(cipher.getAuthTag())
// We set the another associated data, that differs from one used in encryption
let wrongAssocData = Buffer.from('Payment');

// We set the wrong associated data
wrongDecipher.setAAD(wrongAssocData, {
  plaintextLength: ciphertext.length
});

// We start the decryption
let wrongDecrypted = wrongDecipher.update(ciphertext)
// We finalise the decryption and save the plaintext in a variable
try {
  // This leads to exception
  wrongDecipher.final();
} catch (err) {
  throw new Error('Authentication failed!', { cause: err });
}

The code above makes use of chacha20-poly1305 as the AEAD. It provided an associated data. It demonstrates encryption and decryption. It also shows that the decryption will fail, if a different associated data, from the one used at encryption is attempted for decryption.

Introduction to Symmetric Encryption for the Working Developer

noreply@blogger.com (dade) — Sun, 01 May 2022 16:37:00 +0000

Encryption is all about establishing confidentiality. That is, making sure only authorised parties have access to specific data. It involves transforming data into a form that is inscrutable to unauthorised parties but with the ability for authorised parties to transform the data back into its legible form.

Think about two parties communicating without wanting other persons privy to the messages. This is achieved via the cryptographic primitive of Encryption.

In this post, I will provide a whirlwind tour of what encryption is for the working developer. The goal is to provide the basic information needed to be able to properly wield this cryptography tool without going into the inner workings of the algorithm. It is part of the Cryptography101 series of posts.

This post contains the following

What Encryption is and its main components
What Encryption is not
Symmetric Encryption vs Asymmetric Encryption.
Block Cipher vs Stream Cipher.
Overview of encryption Standards
Components of AES
- Key
- Padding
- Mode

Encryption and its main components

Encryption is the part of cryptography that deals with ensuring confidentiality of data. This usually involves taking data and transforming it into what looks like gibberish to unauthorised parties.

Encryption can be used as part of secured communication, where two parties communicate without any eavesdropper being able to make sense of the communication. It is also used to encrypt data at rest, for example the data on a harddisk; making sure it is only accessible by the owner or any other authorised parties. This can also be seen as securing communication, where the other party is the future self that needs to access the content of the harddisk.

It should be noted that encryption must also include the reversible component; that is decryption. This is the process of taking the gibberish output of the encryption process and transforming it back to the original data. Without being able to decrypt, encryption becomes useless.

Both the encryption and decryption process makes use of a critical component called the secret key. The secret key together with the encryption/decryption algorithm is what is used to transform a message into gibberish and also to recover the original message. As long as an eavesdropper does not know the secret key, there is no way to access the message. Hence the secrecy of the key is at the heart of the security of encryption.

The encryption, decryption and secret key generation algorithm is oftentimes together referred to as Encryption Scheme.

The data to encrypt is usually called plaintext, the algorithm that encrypts the data is usually called cipher and it makes use of a secret component called key. The encrypted data is usually called ciphertext.

What Encryption is not

Just as it is important to say what encryption is, it is also important to make clear what encryption is not.

Cryptographic Hashing is not Encryption

Cryptographic Hashing is another cryptography primitive that is sometimes confused with encryption. A cryptographic hash algorithm takes data and produces a random bit string, often called digest, which serves as a form of fingerprints of the data.

Hashing is a one way operation, that is, with a secure hashing algorithm, it should be practically impossible to retrieve the original data from the digest. This is in contrast with what you want with encryption. With encryption you need to be able to retrieve the original message.

See Introduction to Cryptographic Hash Functions for the Working Developer for more about Cryptographic Hash Functions.

Encoding/Decoding is not Encryption

Encoding/Decoding schemes like base64 or even representing data in hexadecimal or binary string is not encryption even though they might also appear to be hiding data. They are not encryption because they do not make use of any secret key, hence they provide no confidentiality since anyone can perform the reverse decoding operation on the encoded data.

Steganography is not Encryption

In infosec there is sometimes the need to hide the fact that communication between two parties is taking place. This usually involves hiding a secret message in something that is not secret. This is done via Steganography. This is different from Encryption which does not seek to hide that communication is taking place but only to hide the message that is being communicated.

With Steganography an eavesdropper won't be aware that communication is taking place.

With encryption an eavesdropper can observe communication taking place but won't be able to see what is being communicated.

So having touched on a couple of things that are not encryption, let us go on to see some different aspects of encryption that are often used to put them into different categories or types.

Symmetric Encryption vs Asymmetric Encryption

As mentioned earlier, the process of encryption depends on the usage of a secret key. You then have two flavours of encryption depending on how this secret key is used.

An encryption scheme where the same key is used for both encryption and decryption is called symmetric Encryption. While an encryption scheme where the key used for encryption is different from the key used for decryption is called asymmetric encryption (also known as asymmetric cryptography or public key cryptography).

This post is about symmetric encryption

Block Cipher vs Stream Cipher

Encryption schemes can also be categorised based on how the plaintext is passed on to the encryption algorithm.

Encryption schemes that require the plaintext to be encrypted a fixed size at a time are called Block ciphers.

This involves having the plaintext broken down to a certain fixed bit size, and each bit size encrypted. This is a hard requirement, hence if the plaintext or the last piece of the chopped up plaintext is not up to the required fixed size, it would have to be padded till it meets the required size.

Stream ciphers, on the other hand, can encrypt plaintexts at varied bits at a time. Since they do not require a fixed block to be passed in for encryption, stream ciphers hence do not have a need for padding.

This post is about symmetric encryption that are block ciphers.

Overview of Encryption Standards

As stated in Introduction to Cryptographic Hash Functions for the Working Developer there are different agencies that publish cryptographic related standards. In this post, we only limit to standards published by National Institute of Standards and Technology (NIST).

When it comes to symmetric encryption, the go to encryption scheme as standardised by NIST is Advanced Encryption Standard, usually referred to as just AES.

As with most standards by NIST, what later came to be the AES was as a result of an open competition that started in 1997. The competition was an open call to all interested cryptographers to come up with an encryption scheme to replace the then standard scheme: Data Encryption Standard usually referred to as DES.

In 2001, the competition was concluded and out of 15 different submissions, the Rijndael encryption scheme, designed by Vincent Rijmen and Joan Daemen emerged the winner and was named the AES.

AES has since then been the main encryption scheme. Replacing DES, which for all intents and purposes is broken and should not be used. Hence the remaining part of this post would be basically an overview of AES.

Components of AES

In this section, I will go over some of the key components of AES. This are key size, padding schemes, and mode of operation.

Keys in AES

Like any symmetric encryption scheme, AES also requires a secret key. AES comes in 3 different versions depending on the size of the key.

We have:

AES-128 takes a key of 128 bits (16 bytes)
AES-192 takes a key of 192 bits (24 bytes)
AES-256 takes a key of 256 bits (32 bytes)

The size of the key dictates the level of security. Larger keys provide higher security but for almost all use cases, AES-128 will suffice.

AES keys are random bits. That means any method that can generate true cryptographic random numbers of 128, 192, and 256 bits can be used to generate AES keys. As mentioned earlier on, the security of AES depends on the key, hence the methods used to generate the key needs to be truly random and unguessable.

For example, in JavaScript environment with Web API available, like in the browser or Deno, window.crypto.getRandomValues(new Uint8Array(16)) can be used to generate random 256 bits that can be used as a AES-256 key.

In Deno, this look like:

Deno 1.16.4
exit using ctrl+d or close()
> let counter = window.crypto.getRandomValues(new Uint8Array(16));
undefined
> console.log(counter)
Uint8Array(16) [
  189,  59,  14, 161,  84,
  117, 160, 119, 175,  96,
  130, 233, 121,  99, 176,
  240
]

Padding scheme in AES

AES is also a block cipher, hence it can only encrypt a fixed size at a time. AES block size is 128 bit hence it can only encrypt plaintext 128 bit at a time.

This means if your plaintext is larger than 128 bit, it would have to be broken up into blocks of 128 bit each. If the plaintext is less than 128 bit, or after breaking into 128 bits blocks, there remains a last piece that is less than 128 bit, then that piece needs to be padded somehow to become 128 bit.

Note that the block size of 128 bit is independent of the key size. Meaning that even if a key size of 192 bits or 256 bits is used, the encryption/decryption would still be done 128 bits at a time.

The padding scheme needs to be such that upon decryption the padding portion can be removed and discarded to reveal the original plaintext.

Hence AES needs a padding scheme. One of the popular standardized padding scheme, which is also used in AES implementation is PKCS#7. Simply put the PKCS#7 algorithm states the value of each padding byte should be the length of the padding required. So in the case where you have a plaintext of 12 bytes, this will require 4 bytes of padding to make it 16 bytes (that is to make it 128 bit that is required by AES). Hence the value of the 4 bytes should be 4.

To prevent ambiguity in the case where the plaintext is already a multiple of 128 bits, PKCS#7 stipulates that we add a full block of padding set to the value 16. This ensures that, to remove the padding, one can always look at the last byte, and the value found there, is the number of bytes to discard in order to get to the main data.

Most implementations of AES do not make the padding scheme configurable to the developer, hence you won't need to fiddle around with the padding scheme most of the time. Irrespective it is good to know, to have a better understanding of AES.

Mode of operation

The consequence of AES being a block cipher, that is being able to only encrypt 128 bits at a time leads to two things: We need to provide a padding mechanism, which was mentioned in the previous section.

We also need a way to put the pieces of encrypted 128 bits together to get the final ciphertext. The way by which the individual encrypted 128 bits are assembled together is called Mode of operation and there are various ways to go about this.

There are various modes that have been defined for block ciphers (i.e. not limited only to AES), but to have a better understanding of what they are actually supposed to help with, we look at two modes: ECB mode and CBC mode.

ECB - Electronic Code Block - Mode

A naive way to piece back the encrypted blocks, is to assemble them back exactly the way the plaintext was broken up. This mode of encryption is called Electronic Code Book (ECB).

The ECB mode has the drawback that it leaks information about the plaintext.

If the plaintexts are broken apart, encrypted and assembled back the same way, patterns about the data can still be carried over into the encrypted version, because similar bits in the plaintext would encrypt to similar cipher text. This allows some information to still be learnt about the plaintext, which defeats the purpose of encryption. A good encryption scheme should not reveal any useful information about the encrypted message, none at all.

A very good example of illustrating this weakness of the ECB mode is the ECB penguin. An image of a penguin is encrypted using the ECB mode, but the shape of the penguin can still be seen in the encrypted image.

Hence for all intents and purposes, ECB mode is usually not used.

CBC - Cipher Block Chaining - Mode

The Cipher Block Chaining mode is another mode of operation that can be used with AES that does not suffer from the weakness of ECB.

The problem with the ECB mode is that identical portions of the plaintext would encrypt to identical cipher text. This is what enables patterns to still be carried over to the ciphertext.

The CBC mode avoids this problem by including a procedure in its algorithm that first randomises the plaintext before encryption. This process of randomisation ensures that identical portions of the plaintext will never encrypt to the same ciphertext.

Simply put, CBC achieves this randomisation by taking a previous encrypted block and using this to randomise (by performing an XOR operation) the next block to be encrypted. This then provides the output that is then fed as the next block to the encryption. This chaining goes on until the plaintext is fully encrypted into the ciphertext.

You might ask, how will the first block be randomised before encryption since there is no previous cipher text that can be used for the XOR operation? The answer is that CBC used with AES requires an additional 16 bytes value called the Initialisation vector (IV) to kickstart the encryption process. This is the value that is used to randomise the first block to kickstart the encryption process. It should be noted that the IV also needs to be random and unpredictable in order to have a secured encryption scheme.

So far we have seen what encryption is, what it is not, the difference between symmetric encryption and asymmetric encryption and also the difference between block ciphers and stream ciphers. We also saw how block ciphers lead to needing padding and mode of operation. The weakness of ECB mode and how CBC avoids that.

Next we will see how encrypting data with CBC looks like in code. To do this, we would again be using Deno.

The following code shows how to encrypt and deprect a piece of text using AES with the CBC mode.

// We generate a key
// See https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/generateKey
let key = await crypto.subtle.generateKey({
   name: 'AES-CBC',
   length: 128
}, false, ['encrypt', 'decrypt']);

// We generate random 16 bytes to be used as initialization vector
let iv = new Uint8Array(16);
// and fill it with random numbers
// see https://developer.mozilla.org/en-US/docs/Web/API/Crypto/getRandomValues
await crypto.getRandomValues(iv);

// The web crypto requires data to be in ArrayBuffer
let plaintext = new TextEncoder().encode("hello world");

// The encryption and decryption requires specifying
// the algorithm and initialization vector
let param = {
   name: 'AES-CBC',
   iv: iv
};

// We encrypt the plaintext to get the ciphertext
let ciphertext = await crypto.subtle.encrypt(param, key, plaintext);

// We decode and console logged the ciphertext
console.log(new TextDecoder("utf-8").decode(ciphertext))

// We decrypt the ciphertext back to plaintext
let deCryptedPlaintext = await window.crypto.subtle.decrypt(param, key, ciphertext);

// We tranform the plaintext from bytes to string and then print it out
console.log(new TextDecoder("utf-8").decode(deCryptedPlaintext));

If you copy this code into a filename enc_dec_aec_cbc.ts and then execute deno run enc_dec_aec_cbc.ts, you should see two lines printed out. One looks like gibberish binary blob, while the second line should be "hello world"

As the comment in the code shows, we are generating a key, and IV, and then used to encrypt the text "hello world". Then print the cipher text and then decrypt it back to the plain text "hello world".

Hash Function in Action: Message Authentication Codes

noreply@blogger.com (dade) — Sat, 23 Oct 2021 16:20:00 +0000

In Introduction to Cryptographic Hash Functions for the Working Developer, I presented a straight to the point, overview of some essential things a developer should know about cryptographic hash functions.

This post continues in the theme around hash functions, by taking a look at another cryptographic construction hash functions make possible, that is: Message Authentication Codes (MACs).

It is worth quoting Bruce schneier again:

Much more than encryption algorithms, one-way hash functions are the workhouses of modern cryptography

Because Message Authentication Code based on hash functions is a perfect demonstration of how crucial hash functions are.

This post contains the following sections:

Why Hash Functions alone are not enough
What is Message Authentication Code
What is a Hash Based Authentication Code (HMAC)
What is a Keccak Based Authentication Code (KMAC)
Some real world applications of Message Authentication Code

Hash Functions alone are not enough

Hash functions produce random fixed length finger print of input data. As it turns out, this construction alone by itself, is hardly ever enough for any real life use. This is because just hash functions alone are not enough to guarantee things like integrity and authentication. Things hash functions would be used for.

To illustrate this, take browser cookies for example. There is the need to check the integrity of cookies. To ascertain that cookies placed with a browser by a server, is the same returned by the browser on consequent requests. A naive way to check this, is to just use hash functions: hash the contents of the cookie and place it on the browser alongside the data.

This has a loop hole, in the sense that an attacker can intercept the request back to the server, modify the cookie data, generate another hash and send both back to the server. The server won't be able to detect there has been a modification.

This shows, how in most use cases, just a hash function is never enough. In the scenario described, the loop hole is due to the fact that hash functions are public and hence anyone can use them. Therefore we need a way to include something not public, something secret into the mechanism that can be used as a way to authenticate the output of the hash function. Message authentication code (MAC) are constructs that can be used for this.

It turns out that hash functions can be used to generate these MACs when used together with secrets.

Understanding what this construction is all about is the goal of this post and we start with first looking at Message Authentication Code.

What is Message Authentication Code

A message authentication code, is extra data or information that is used to confirm that a piece of data came from the stated sender (confirming authenticity) and has not been changed (providing integrity). This piece of extra data/information is often referred to as the authentication tag.

In general authentication tag can be broadly classified into 4 categories: unconditionally secure, hash function-based, stream cipher-based and block cipher-based. In this post, I would only be touching on two of the hash function-based: HMAC (Keyed-hash message authentication code) and KMAC (KECCAK Message Authentication Code).

In summary, the hash function based MACs can be seen as a mix of a hash function and a secret. Let's look first at HMACs.

What is HMAC

HMAC is a MAC created from using a cryptographic hash function and a secret cryptographic key.

The cryptographic hash function can be any secure hash function, such as SHA-2 or SHA-3. This is reflected in the name of the given HMAC. For example, HMAC-SHA256 and HMAC-SHA3-512 is created using a secret key with SHA-2 (256) and SHA-3 (512) respectively.

As noted in Introduction to Cryptographic Hash Functions for the Working Developer naively using SHA-2 to hash concatenated secrets with a message is insecure due to the weakness of SHA-2 to the length extension attack. This is why HMAC is needed when the requirement is to use SHA-2 with a key in other to create an authenticated tag.

Although it is beyond the scope of this post to dig into how HMAC works, it is worth pointing out that HMAC construction is not a naive concatenation of secret keys and data. HMAC uses a nested construction which avoids the length extension attack pitfall, Hence it is a special one that takes care of combining secret keys and data together in such a way to avoid being susceptible to the length extension attack.

SHA-3 on the other hand is not susceptible to the length extension attack and it is relatively safe to create an authentication tag manually using SHA-3-512(𝐾𝐸𝑌‖𝑚𝑒𝑠𝑠𝑎𝑔𝑒), but since the HMAC construction is available, it is also possible to use the construction with SHA-3 instead, hence HMAC-SHA3-512.

Using HMAC-SHA3-512 is not advised though, as it is not efficient. As stated in NIST SP 800-185

KMAC is a keyed hash function or pseudo-random function (PRF) that can be used, e.g., to compute a message authentication code (MAC) or to derive a session key from a master key. It is more efficient than HMAC by removing the need for HMAC's nested construction

Hence when there is the need to use SHA-3 hash functions to create authentication tag, it is preferable to do so using KMAC, which we look at next.

What is KMAC

The KECCAK Message Authentication Code (KMAC) algorithm is also a keyed hash function but based on KECCAK.

It provides variable-length output, and unlike SHAKE and cSHAKE, altering the requested output length also generates a new, unrelated output.

KMAC has two variants, KMAC128 and KMAC256, built from cSHAKE128 and cSHAKE256, respectively.

Some real world applications of Message Authentication Code

JSON Web Token

The JSON Web Token: JWT is an open, industry standard for representing claims securely between two parties. It guarantees integrity and also authenticity. For example JWT can be used to solve the problem of servers being able to verify the integrity and authenticity of the cookies they receive browser clients.

JWT provides the option of specifying HMAC construction as a signing algorithm.

See here and here for more information on JWT.

HMAC-based one-time password

A One Time Password (OTP) is a dynamic password that is valid for only one login session or transaction, or a short period of time. Certain implementations of OTP makes use of HMAC. These are called HOTP: HMAC-based one-time password. A popular example of such an HOTP is the Google Authenticator

Signing HTTP Requests

HTTP links used for sensitive operations like password resets or payment links can be accompanied with MACs. This helps in ensuring the integrity and authenticity of such links.

Introduction to Cryptographic Hash Functions for the Working Developer

noreply@blogger.com (dade) — Wed, 20 Oct 2021 18:49:00 +0000

Much more than encryption algorithms, one-way hash functions are the workhouses of modern cryptography - Bruce schneier

This post would be a quick, straight to the point, overview of some essential things a developer should know about cryptographic hash functions. It is targeted at the working developer who needs to be familiar enough with cryptographic hash functions in order to use them, but who does not need to know the gory details of how they are implemented, all their possible use cases or how they work internally.

This post contains the following sections:

Cryptographic hash function: A definition
Properties of cryptographic hash functions.
Types of hash functions

Fixed length hash Functions
Extendable Output Functions (XOF)
Password hashing functions

Some Hashing Hygiene

Cryptographic Hash Function: A Definition

A hash function is a function that converts any arbitrary data into a random, fixed sized data. One can see it as a way to generate a unique fingerprint for any arbitrary data. The output of a hash function, the generated fingerprint, is usually referred to as hash values, hash codes, digests, or simply hashes.

For example, using OpenSSL, generating a digest for the phrase "hello world" using the Sha256 hashing algorithm will look like this:

> echo 'hello world' | openssl dgst -sha256

a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447

The `a94890…` is the digest of the string "hello world".

A somewhat similar looking output, similar in the sense of it being unique and random looking, can also be generated using the `cksum` utility

> echo 'hello world' | cksum

3733384285 12

What then is the difference between the output of the cksum utility and that of sha256 using the openssl cli?

They are both hash functions, but sha256 is the one that can be described as a cryptographic hash function, while cksum is, well, just a hash function. You also have hash functions that are not cryptographic hash functions used for instance in data structures like Hash map. Being able to turn an arbitrary data into some sort of unique finger prints is not enough to make a cryptographic hash function.

The question then is, what makes a hash function a cryptographic hash function?

For a hash function, to be actually considered a cryptographic hash function, it has to satisfy certain properties. These properties are what we would look at next.

Properties of cryptographic hash functions

For a hash function to be a cryptographic hash function it most posses the following properties:

Pre-image resistant

This property guarantees it should be infeasible to get the original data that produced a particular hash. A less fancy term, it is to say the hash function should be one-way. You should only be able to go from the data to the hash value, and there should be no practical way to determine the original date from the hash value.

Second pre-image resistant

This property guarantees if you have a data and it's hash, it should be infeasible to come up with another data that hashes to the same value as the previous data. Expressed In a more mathematical format: it says given x (original data) and h(x) (hash of original data), it is infeasible to find y (another data) such that h(y) = h(x). This property is also called Weak collision resistance.

Collision resistant

This property guarantees that it should be infeasible to come up with two different data that hashes to the same value. Again, in a slightly mathematical format: it should be infeasible to find any x and y, with x != y such that h(x) = h(y)

Avalanche effect

This property guarantees that the output of a cryptographic hash function is as random as possible and bears no semblance or connection to the input data. The avalanche effect ensures the output changes significantly if the input is changed slightly.

> echo raw | openssl dgst --sha256

SHA2-256(stdin)= 8e5ceeca3a438135cfd1372eafe969ccc4440798e378d8b8ed24242f026a704f

> echo rat | openssl dgst --sha256

SHA2-256(stdin)= fd439a2284b3e6aaa3c0b6d8fc21142ce0398dc42d12315165e22908aadca92f

A single change in the input: from raw to rat, changes the output significantly. This is the avalanche effect.

These properties, if present, means you have a cryptographic hash function and not just an ordinary harsh function.

Next we would take a quick look at how one might categorise cryptographic hash functions based on their types/usage.

Types of hash functions

There are different agencies that publish cryptographic related standards. In this post, we only limit to standards published by National Institute of Standards and Technology (NIST). Curious readers, with the help of their favourite search engine can find similar bodies, that for example are responsible for cryptographic standards in China, Israel etc. Having said this, let's take a look at some of the categorisation.

In this section, we look at: Fixed size hash functions or message digests, XOF: Extendable Output Functions and Password hashing functions.

Fixed length Hash Functions

The hash functions listed in this section are the ones that take an arbitrary length of data and always produce a fixed length digest as output.

SHA-2

SHA-2 stands for Secure Hash Algorithm 2 and was invented by NSA and standardised by NIST in 2001. SHA-2 provides 4 different versions each producing different bit sizes. The bit sizes are 224, 256, 384, and 512 bits, and named SHA-224, SHA-256, SHA-384, and SHA-512 respectively. (Note that the names omit the version).

SHA-2 is a Merkle–Damgård construction. No need to worry yet what this means. This detail is only mentioned now, as it will be used to make the point on how SHA-3 is further different from SHA-2.

SHA-3

SHA-3 is another family of hashing algorithms that was standardised by NIST. It came about from a competition that was initiated by NIST in 2007 to find a replacement for SHA-2. One of the submissions, Keccak, emerged as the winner and took the name SHA-3.

SHA-3 offers the same variants as SHA-2, but has the full name in their named variants: SHA-3-224, SHA-3-256, SHA-3-384, and SHA-3-512. In 2015, SHA-3 was standardised in the

FIPS Publication 202.

It should be noted that the padding parameters defined in original Keccak was modified in the resulting SHA-3, hence it is possible to have cryptographic libraries that offer Keccak implementations whose digest will be different from what SHA-3 outputs. This padding difference is the only thing distinguishing Keccak and SHA-3. They both offer the same level of security.

SHA-3 is built with a sponge construction, which is different from Merkle–Damgård construction which SHA-2 is based on.

SHA-1 and MD5

SHA-1 and MD5 are also standardised fixed length hash functions. You should not be used though, because they have been found to be broken. They are mentioned here for completeness sake.

One interesting thing to note is that both SHA-1 and MD5 are also based on the Merkle–Damgård construction. Keccak, from which SHA-3 is derived, uses a sponge construction, which differs from the Merkle–Damgård construction. This was one of the reasons why it was chosen as the winner of the SHA-3 competition. Rationale is if Merkle–Damgård construction is totally broken, then SHA-3 would be spared.

BLAKE

Another popular hash algorithm is BLAKE (and the subsequent improvements: BLAKE2/BLAKE3).

BLAKE made it to the final round of the NIST competition that produced SHA-3, but did not end up being the winner.

One of the reasons why it was not the winning submission is due to the fact that it makes use of the Merkle–Damgård construction, which sort of defeats the purpose of the SHA-3 competition, which is to come up with another hashing standard that won't be susceptible to the same issues as SHA-2.

XOF Extendable Output Functions

Extendable-output functions (XOFs) are a family of cryptographic hash functions that can output digests of arbitrary size. This is in contrast with the fixed length hash functions that can only output digest of specific sizes.

These XOF can be used to create other cryptographic constructions like random numbers, as key deriving functions (KDF).

SHAKE

SHAKE (Secure Hash Algorithm Keccak) is an XOF that was included in the standardisation of SHA-3, as published in FIPS Publication 202.

One can think of SHAKE as being exactly like SHA-3 only with the additional option of specifying how long the digest should be.

cSHAKE

This is a customisable SHAKE. It is just like SHAKE, in the sense that it allows for the creation of variable sized digest. The difference between cSHAKE and SHAKE is that cSHAKE allows for customisation via an input string.

So with cSHAKE, you can not only generate a varied length digest for an input, the digest can be further customised based on an optional input string passed to the function.

cSHAKE is part of SHA-3 derived functions; a set of functions that was published a year after SHA-3 was standardised in the Special Publication 800-185.

TupleHash

TupleHash is also another XOF, in that it is a variable-length hash function designed to hash tuples of input strings. This hash function should be used in scenarios where a list of data needs to be concatenated together before hashing.

To illustrate why a hash function like TupleHash is needed, imagine a string of the format below:

Which represents a transaction to pay at a restaurant. A concrete example of such a string and its hash can be seen to be:

> echo "Joe""Fibo""50""20" | openssl dgst -sha256

SHA2-256(stdin)= bb7125e94fcf1df473523022d0d5232c1f45580821bd68c9d01a072004599e44

The same string can be modified, such that the payment becomes 502, and tip becomes 0 and it will still hash to the same output

> echo "Joe""Fibo""502""0" | openssl dgst -sha256

SHA2-256(stdin)= bb7125e94fcf1df473523022d0d5232c1f45580821bd68c9d01a072004599e44

It is for situations like this, that TupleHash should be used.

TupleHash is part of SHA-3 derived functions; a set of functions that was published a year after SHA-3 was standardised in the Special Publication 800-185.

ParallelHash

ParallelHash is another XOF, in that it is a variable-length hash function that can be used to hash very long messages in parallel.

ParallelHash is part of SHA-3 derived functions; a set of functions that was published a year after SHA-3 was standardised in the Special Publication 800-185.

Password Hashing Functions

There are cryptographic hash functions that are designed specifically to hash passwords. These are hash functions designed to be used together with salts and also slow to deter brute force attacks. Some hash functions that fall under this category include: Argo2, PBKDF2, bcrypt and scrypt.

Some Hashing Hygiene

To wrap up this post, I'll outline some common security precautions to keep in mind when using cryptographic hash functions. This is in no way exhaustive, but should still be useful.

Don't use MD5 and SHA-1

These two algorithms are no longer considered secure. Even though all currently known attacks on MD5 and SHA-1 are collision attacks (i.e. they are still resistant to pre-image and second pre-image attack), you are better off using newer hashing algorithms.

Don't use SHA-2 to hash secrets due to length extension attack

There is sometimes the need to append a secret key to message being hashed. This is often done to achieve some level of authentication. Hashing this way should never be done using SHA-2.

Even though SHA-2 is a perfectly fine hash function to use, it is not suitable for hashing message that contain appended secrets. This is because of a downside of the Merkle–Damgård construction, which makes SHA-2 vulnerable to length extension attack.

Explaining the length extension attack is out of scope of this post, but it worth mentioning that SHA-2 should not be used in situations, where a secret is to be included in data to be hashed. For such use cases, HMACs should be used. HMACs would be covered in another post.

Minimum output size of hash function should be 256 bits

To reduce the feasibility of a successful collision attack, the minimum output size a hash function must produce should be 256 bits. This ensures a minimum of 128 bits of security. A 128 bits of security means an attacker needs to perform 2 raised to power 128 operations to be exhaustive, which is a huge number. The reason why an output of 256 bits is required to have a 128 bits of security is due to something called Birthday problem. This has roots in probability theory but explaining it, is out of scope of this post.

Don't hash small or predictable values

For example, if it can be predicted that the value that is being hashed is either yes or no, then it would be trivial to hash all possible values suspected to compared to the hash. Technically making it possible to derive the input data from the hash.

Don't naively hash concatenated inputs

As shown in the section that motivated the use of TupleHashes, naively concatenating strings might lead to unexpected security loop holes.

NFTs are not to blame

noreply@blogger.com (dade) — Tue, 28 Sep 2021 16:29:00 +0000

T206 Honus Wagner baseball card sold for $6.606 million just this August 2021, Pikachu Illustrator, a pokeman card sold for $195,000 in 2019. You can get different Micheal Jordan's basketball card on e-bay ranging from couple of thousands to couple of millions. Sentimental and collectible items have always been part of the human experience. NFTs are not to blame.

Why should pieces of cards with pictures of athletes (or game characters) be worth that much? Why should JPEGs/PNGs be worth that much? Maybe because humans also value things for sentimental reasons just as they do for utilitarian/sustenance reasons.

NFTs only made the act of ownership that was so easy to establish in the physical realm, also possible in the digital realm. It's only natural it will be used for digital arts/collectible culture.

The mistake to avoid, is thinking NFTs are only about digital arts and collectibles. No! they are more than this. The ability to create unique, non fungible digital items that can be provable owned, can be, and is being deployed for other use cases.

Why I am offering NFTs as a crowdfunding perk

noreply@blogger.com (dade) — Mon, 13 Sep 2021 04:36:00 +0000

Non-equity funding campaigns are always based on promises: "Back us now, and we promise you early access to the product when we launch, or we promise you a discount, digital shout-out or swags". I am currently running an Indiegogo campaign where the promise will also include an NFT: Non Fungible Token, which is a digital artifact that can be used to prove things like ownership. This, as far as I know, is the first time NFTs would be used in this capacity.

In this post, I will shed some light on what NFTs are and share my thoughts on why they are a perfect perk for a non-equity crowdfunding campaign. In doing so, I will touch on my thoughts on the continual march towards the digitalization of the human experience. How digitalization and technologies like virtual/augmented reality, blockchain technology, etc are making it possible to further digitize aspects of society and human interaction that were previously rife with problems or just plain impossible to digitize.

By the way, my campaign is to kickstart the development of DishAfrik: a recipe app to discover, enjoy and share African recipes. Think of it as Yummly or Tasty.co but focused on African cuisine. One of the perks of the campaign is going to be 54 uniquely generated collages made up of 54 recipes from 54 countries. 54 because currently Africa is home to 54 fully recognised sovereign states.

You should check out the IndieGoGo campaign and back the project :) Find it here

TL;dr

Ever since the third industrial revolution, there has been a march towards digitalization of the physical world.
Blockchain-related technologies, like NFTs, provided the possibility to digitize aspects of the physical world that relied on scarcity, uniqueness, and proof of ownership, aspects that were previously hard to digitize.
DishAfrik's Indiegogo campaign will be making use of NFTs as a perk, using it as a way to capture participation, while also creating digital art in the form of a recipe collage that can also be owned.

In the beginning, before the Metaverse

It started with the Third Industrial Revolution, often referred to as the Digital Revolution. A period that began in the late 1900s that led to the spread of automation and digitization through the use of electronics and computers. The technological invention of this era made it possible for things that once existed only in the tactile realm to now also exist in a new and alternative realm: the digital world.

Obviously, such powerful technology would have an impact on society. First, it was radio that killed the town crier, and then video killed the radio star. It was not enough to have good old mails, we got new ones with an e as a prefix: e-mails. The music industry had its Napster scare, and then we had the spotify-cation of music distribution, which killed the CD, at whose hands, cassette and VHS suffered similar fatal fate. Journalism, Medicine, Education, Entertainment you name it, all got impacted by this continual digitalization wave.

Even with all these seismic shifts that digitalization brought, there were certain aspects of society and human interactions that just could not be properly represented in the digital world. This is because these human interactions are inherently at odds with the very nature of digitalization.

For example, digitalization makes it possible to produce digital artifacts once, and then duplicate them till Infinitum with little or no extra costs. Also for all practical purposes, nothing is distinguishing an original copy of a digital artifact from a duplicate.

These are unique characteristics of digitalization. Given this, how then would it be possible to properly digitize commerce for instance? How would it be possible to create e-money just like e-mail? Electronic mail is not problematic, because duplicating a message and sending it multiple times does not in any way compromise the objective of communication which email serves, the same can't be said of a naively designed digital version of money.

If the same endless duplication is possible with digital money, what is stopping someone from making multiple copies of a digital coin, to spend the coin multiple times? Being able to do this negates the ability to carry out commerce in the digital world. This problem, by the way, is called the double-spend problem.

Also, how can things like ownership be properly grafted into this new digital world? Since a digital copy of a digital thing is as good as the original?

Trusted third parties, bottlenecks, and anarchists

To enable commerce, on the internet, humans had to come up with financial institutions that acted as trusted third parties. This is not a new thing. The antithesis of anything digital when it comes to commerce, the paper money, originated in China in the 7th century, on the back of trusted third parties.

Before the use of these notes, the Chinese used circular copper coins. If you were wealthy, these copper coins soon became a burden to carry around, especially when used for larger transactions. To solve this problem, the coins could be deposited with a trusted person, who then gave the wealthy merchant a slip of paper (the receipt) that says how much money they had deposited. This slip of paper was then used in commercial activities in the exchange of goods and services. When the recipient of the paper slip returns to the trusted party, they could then exchange the slip of paper for the actual coins.

In the hundreds of years between 7th century China and the 20th century, human monetary affairs came to follow the Chinese model. Therefore it is no surprise that a model involving trusted third parties was also employed when commerce first moved online. In this model, the trusted party serves as the electronic mint that creates and tracks every digital coin that is being spent. In doing so, they can prove ownership and who is allowed to spend what coin, when it is spent, and that it can't be spent twice.

The first successful foray into digitizing money was E-gold. E-gold was a digital gold currency run by Gold & Silver Reserve Inc, which served as trusted third parties. In this capacity they enabled people to transact business online by moving computer bits from one location to another with the assurance that transaction accounting was handled by the E-gold system and the actual money would be made available to parties when requested.

It is also worth pointing out that trusted third parties are not limited to commerce. They are a common modality when there is the need for an intermediary to mediate verification and exchange of value. Value could be money, or it could be real estate (then you need a notary), or IP addresses (then you need internet registries). The scenarios abound where trusted third parties are relied upon.

So with this model, digitalization of commerce, tracking, and transferring anything of value was made possible. Finally! Well, maybe not quite.

There is something not quite kosher, with a model involving a trusted third party: and that is the need to have a trusted third party. As the saying goes, absolute power corrupts absolutely. Trusted parties usually occupy a unique role that bestows upon them, if not absolute power, the closest thing to it.

Apart from the political schools of thought that would rather get rid of trusted third parties, there are other weaknesses to relying on trusted third parties. For one they are usually a single point of failure, or point of corruption, depending on the situation. They can also be bottlenecks as their inefficiencies can have rippling effects. Trusted third parties are also by their very nature centralized, and centralized systems are generally harder to scale. The internet would never have evolved into what it is today if it was centralized. (The internet is decentralized by the way).

What then do we have? We are left with a deceleration in this march towards the digitalization of everything previously analog. The best we could come up with left us with an internet, where commerce, that quintessential human activity, could not be liberated from trusted third parties. It was also a state where concepts like ownership could not be fully grafted onto the digital world.

Cryptography wins again!

The genius of Satoshi Nakamoto, the creator of blockchain was in conceiving a protocol that relied on cryptographic primitives to mediate the transfer of value without requiring the services of trusted third parties. Instead of trust, the cryptographic proof is relied upon. This creation is often referred to as the blockchain.

The blockchain is often seen as a synonym for bitcoin, which is not surprising, as bitcoin was the first implementation of the blockchain. But the blockchain is more than bitcoin. At the most fundamental level, the blockchain can be seen as a distributed ledger. The protocol outlined by Satoshi Nakamoto made it possible to have a distributed ledger that can track ownership and the transfer of ownership of digital artifacts, without needing a centralized repository. Instead of trusted third parties, a peer-to-peer, decentralized model is used, which relies on cryptographic proofs to keep the integrity of the ledger.

This means, instead of relying on centralized, trusted third parties to keep records of ownership and transfer of ownership of digital artifacts, we rely on cryptography instead. Making it possible to put information in a distributed ledger where everyone and anyone can see its contents, inspect and verify. This means once a transfer of ownership is reflected in the ledger, another attempt to invoke the same ownership transfer would be visible to all relevant parties involved and declined.

This invention made it possible to truly elevate money into the digital realm. Usage of physical money has never had to deal with the problem of double-spending, due to the very nature of it being physical. Once a coin goes from one hand to another, it is impossible for the former holder to still have the exact coin and attempt to transfer it to someone else. This was possible in the digital realm and for a long time, having a mediating third party that serves as a mint, was the only solution. Blockchain created a system that was able to impose constraints similar to those that come with physical money on digital currency, while avoiding the centralization trap. The protocol was able to achieve this without the need to have a trusted third party.

Satoshi Nakamoto's outline of the blockchain was just the spark that ignited other ideas that have led to the emergence of the cryptocurrency/blockchain industry. Once it became clear how to transfer digital artifacts between parties, the next step was to make this programmable. And this is exactly the innovation Ethereum brought into the space.

Ethereum created a computation layer on top of the blockchain, making it possible to encode autonomous rules that control how digital artifacts are to be moved between parties on top of the blockchain. This layer of computation on the blockchain is often referred to as smart contracts. Even though it was coined in the 1990s, by Nick Szabo, arguably Ethereum was what made the term smart contract mainstream.

Another important idea that proved to be critical is the idea of permanence. There are cases where information is required to be unmodifiable. For example, the details of a contract for instance should be unmodifiable after the fact. Unfortunately digital information cannot generally guarantee such a constraint. Digital information is by nature modifiable.

Permanence is one of the central value propositions of the blockchain. If you think about this, it makes sense. A technology that records transactions of value should not allow rewriting of history. Information contained on the blockchain can be considered permanent because the way the protocol is assembled, again making use of cryptography, is such that the cost of going back to change any information on it is so great that it is effectively impossible.

The only snag is, the blockchain is not designed to be a store of arbitrary large data. It does not scale for such a use case. This is where the idea of a permanent web comes into the picture, which, simply put, is a web where contents are immutable. That is, once content is created it should not be editable.

The current web is mutable. Digital content at a particular URL can be changed even though the URL remains the same. Having such mutability is at odds with the need to make statements about permanent events on the web. For example, a contract between two parties found at a particular URL provides less value when there is no guarantee that the content at the URL would always stay the same. A permanent web provides such guarantees.

There has been, and continues to be many attempts at implementing such a permanent web. Arguably, the effort that has been enjoying the most success is IPFS. Which is not only a permanent web but also a distributed web. With IPFS, it is possible to offload large data that needs to be immutable out of the blockchain to the permanent web and have only a reference stored on the blockchain.

The appearance of these technologies: blockchain, smart contracts, permanent web, etc has made it possible for us to continue with the onward march of digitizing our world. These technologies have made it possible to replicate the properties of physical items like scarcity, uniqueness, and proof of ownership in the digital world. Properties that were previously hard to digitize.

Emergence of the NFT

Now that we have these technologies, as is usually the case, thanks to human ingenuity, what has followed is the realization that these technologies can be put to use in clever ways. We have since seen their applications in the financial industry (Defi — Decentralized finance), to new and creative ways organization can be managed (DAO — Distributed Autonomous Organisation), to applications in the supply chain industry.

How blockchain and smart contracts can be used is almost endless and rightly so. This is because a lot of society and human interaction involves dealing with scarcity, uniqueness of things, ownership, and transferring of value. Once we had the means to represent these in the digital world, the possibilities became endless.

A recent application of blockchain and smart contracts that succinctly captures this is the emergence of what is now known as NFTs (Non-Fungible tokens), which have proven to be particularly germane in the creative — music, art, collectible — industry.

What actually is a NFT?

Let's start with what a token is. A token can be said to be a digital artifact that could either exist 100% in the digital world, without any physical counterpart or a digital representation of something physical.

A token could represent currency, lottery tickets, shares in a company, reputation points in an online platform, title deeds, etc.

Tokens could also be of two kinds. They are either fungible or non-fungible. A fungible token is a digital artifact without identity and hence not unique. Non-fungible tokens on the other hand are inherently unique because they possess an inherent identity.

To illustrate: A 5 dollar bill is as good as the next 5 dollar bill. This is because each banknote does not function on the basis of a unique identity, hence they are fungible. Which is another way of saying they are replaceable with one another, as long as their values are the same. Some other artifacts, on the other hand, carry intrinsic identity and can't just be replaced by another. For example a birth certificate, if tokenized, would be non-fungible.

An NFT is then a unique, digital artifact that can exist on a blockchain.

Why NFTs?

It turns out there are a lot of things in the physical world that are valuable because of their non-fungibility. Being able to capture this and represent it in the digital world is what makes an NFT such a valuable piece in the crypto space.

In the digital art sphere, an artist can create a work of art, sell it to a buyer and issue an NFT that can be used to prove ownership of that unique work of art.

In the gaming industry, NFTs can be used to represent and prove ownership of unique in-game items.

In the music industry, an artist can issue a unique NFT to everyone who buys an album. Such an NFT can then be used to prove ownership of purchase and/or to unlock extra content.

NFTs can also be used to create digital collectibles as well as memorabilia, in which case the NFT can be used to capture and document the authenticity of a memorable event.

NFTs can also be used as perks in crowdfunding campaigns. This is exactly what the DishAfrik NFT will do.

DishAfrik's NFT perk

As earlier stated, I am currently running an IndieGoGo campaign to kickstart the development of DishAfrik: a recipe app to discover, enjoy and share African recipes.

One of the perks for backers of the Indiegogo campaign will be an NFT. The NFT will be a uniquely generated image collage similar to Beeple’s Everydays: The First 5000 Days . The NFT will consist of pictures of 54 recipes from 54 African countries. The ingredients and cooking steps will be included as part of the NFTs meta-data. The NFT galley would be hosted on CryptoDisho.

The DishAfrik NFT perk will serve as a means for capturing participation in an activity (in this case the backing of a non-equity fundraising campaign) and encode that in the digital world. Recipients of the NFT perk will be able to prove that they indeed backed the IndieGoGo campaign.

The NFT will also be a digital art that can be owned: a picture made up of a collage of 54 recipes from 54 African countries.

As the IndieGoGo campaign will be an event in time, the NFT will also have memorabilia value. Who knows what this will be worth in the coming years?

Participation in an event, prove-able ownership of art, and memorabilia artifacts were all things that existed without hassle in the physical world before now. But with the advent of new technologies that further the convergence of the physical and digital world, it is now possible to represent these things also in the digital world.

This is why the DishAfrik NFT perk exists.

Metaverse, virtual and augmented reality

This convergence of the physical and digital world that started with the digital revolution shows no sign of abating. One can say we are on the cusp of breakthroughs in these endeavors. Technologies within the space of virtual and augmented reality are exploring ways to further translate other unique physical attributes like "presence" into the digital world.

Mark Zuckerberg thinks this is the future. And spurred on by this belief he has laid out his intention to transform Facebook into a metaverse company. This would turn Facebook into a place where there is the convergence of physical, augmented, and virtual reality in a shared online space.

Although we do not have such a metaverse yet, and it might take a couple more years before its full realization, we do have technologies like NFTs now. And because of that, there is a campaign to build a platform to showcase African cuisine that if you back, you have an option of owning an NFT.

So you should go check that out now!

Understanding the Magic behind Utility Types in TypeScript

noreply@blogger.com (dade) — Fri, 11 Jun 2021 13:50:00 +0000

This post will be the conlcuding post in the Introduction to Advanced Types in TypeScript series. It looks at some of the utility Types in TypeScript, explain how it works while pointing out the features within the TypeScript type system that makes it possible.

Why TypeScript?

TypeScript comes with a powerful and expressive type system. Its expressiveness makes it a joy to work with while its powerful features makes it possible to build, and scale large codebases by providing type safety.

One of the ways TypeScript brings about type safety is by providing developers the tool to manipulate types in ways that encode constraints within the type system. This then ensures that code that could lead to runtime exception do not compile, making it possible to catch errors during development time and not in production. One of such tools TypeScript provides for this are Utility Types.

Utility Types are a set of Generic Types that come natively within TypeScript that makes it possible to transform one type into another. This sort of type transformation is useful because it makes it possible to take exiting types, and apply modifications to it which would ensure the enforcement of certain constraints.

In this post, we would look at 2 of such Utility type and how they could be used to provide more type safety. After that, we would take a step back to understand some of the other TypeScript features that come together to make Utility Types possible. Armed with this knowledge, we will then demystify Utility types by taking a peek under the hood of the 2 Utility types in focus to see how they are implemented.

The two utility types that will be exmaned are Required<T> and Exclude<T>. This post assumes basic knowledge of TypeScript and what Generic Types are. Other features of TypeScript type systems would be explained.

Required<T> and Exclude<T>

We first look at how to make use Required<T> and Exclude<T>. Starting with Required<T>.

Using Required<T>

Required<T> is an utility type that creates a new type based on an existing one by marking all the properties in the new type required.

To illustrates let’s imagine we have a Post type. A post can also have a score value which is calculated dynamically or might not even be calculated yet, hence the score property on the Post type should be optional. The definition of such a post type could look like this:

type Post = {
 title: string
 publishedDate: number
 score?: number
}

When the score on a post has been calculated we want this to be obvious and explicit. Having score as optional does not give use explicitness. One way to go about this, is to create another type with score not optional. For example:

type ScoredPost = {
 title: string
 publishedDate: number
 score: number
}

And we can then have a function that scores a post by taking a value of Post and converting it to a value of ScoredPost

function scorePost(post: Post): ScoredPost {
   return {
      title: post.title,
      publishedDate: post.publishedDate,
      score: post.publishedDate * Math.random()
     }
}

But having both Post and ScoredPost is a bit verbose as it is clearly obvious that ScoredPost is just a Post with the score property now required.

Having Post and ScoredPost is not only verbose it also adds to maintenance burden as any update to either of the types need to be manually reflected on the other.

Since ScoredPost is clearly a transformation of Post with score now required, we can easily apply the Required<T> utility type here. To do that, the scorePost function would be updated as follows:

function scorePost(post: Post): Required<Post> {
    return {
      title: post.title,
      publishedDate: post.publishedDate,
      score: post.publishedDate * Math.random()
     }
}

This shows the power of a utility type such as Required<T>: easily create a new type from an existing type while enforcing the required property constraint.

Using Exclude<T, U>

Exclude<T, U> is also another utility type that creates a new type based on an existing one. It does this by excluding certain properties from one type to create a new type.

To illustrates let’s imagine we have a AdminRights type and a UserRights type. The UserRights could either be read or write, while AdminRights is all the permissions presents in UserRights with the execute permission added. These two types could be defined as follows:

type  AdminRights = "read" | "write" | "execute"
type  UserRights = "read" | "write"

But having both AdminRights and UserRights can be seen as being verbose since UserRights is AdminRights without the execute option.

It also does not make it explicit that UserRights is all the AdminRights with the execute excluded. To make this clearer we can use the Exclude<T, U> utility type.

The definition would then look as follows:

type AdminRights = "read" | "write" | "execute" 
type UserRights = Exclude<AdminRights, "execute">

Here, we are creating a new type UserRights, from AdminRights by excluding the "execute" property. Making it explicit how UserRights is related to AdminRights.

Now we have seen two examples of Utility types in TypeScript. We would now take a quick look at some of the features of TypeScript that makes these utility types possible.

Building Blocks of Utility Types

KeyOf

The KeyOf type operator is an operator that can be used to get all of the properties of a type as a union type. For example, given the type Post:

type Post = {
 title: string
 publishedDate: number
 score: number
}

All the properties of Post, that is title, publishedDate and score can be retrieved as union type using the keyOf operator:

type Props = keyof Post
// Props => "title" | "publishedDate" | "score"

Check Introduction to Index Types in TypeScript for a more detailed introduction to the keyof operator.

Conditional Types

Conditional types can be thought of as a mechanism by which we can perform ternary operations on types. For conditional types, the operation that determines which type is returned is an assignability check, unlike a boolean check as is the case with normal ternary operation that works on values. A conditional types look as follows:

type  Post = {
   title: string
   publishedDate: number
   score: number
}
// the conditional type
type  RatedPost<T> = "score"  extends  keyof  T ? T : never

In the above code snippet, RatedPost\<T> is defined as a conditional type, which makes use of generics. It returns the type T as long as it has the score property, if not it returns never. never is a type that has no value ever, hence if we have a compile error if the conditional type ever evaluates in such a way that the never is returned.

See Conditional Types in Typescript for a more indepth exploration of Conditional Types. See any, unknown and never types in Typescript for a more detailed introduction of the never type

To see this in use, we can define a function: resetScore that only takes a post that already has a score. Such definition would look like this:

function resetScore(post: RatedPost<Post>) {
  // do stuff
}

This can be called as follows:

resetScore({
 title: "hello post", 
 publishedDate: 1621034427002, 
 score: 2
})

but passing an object that has no score property would lead to a compile error:

resetScore({
  title: "hello post", 
  publishedDate: 1621034427002
})

Argument of type ‘{ title: string; publishedDate: number; }’ is not assignable to parameter of type ‘Post’. Property ‘score’ is missing in type ‘{ title: string; publishedDate: number; }’ but required in type ‘Post’

Mapped Types

A mapped type is a type that is created by using the keyof operator to access the properties of another type and modifying the extracted properties in some way to create a different type. For example, a Mapped type that takes a Post type and convert all its properties to string would be defined as follows:

// initial type
type  Post = {
  title: string
  publishedDate: number
  score: number
}

// mapped type
type  StringPost<T> = {
   [K  in  keyof  T]: string
}

// usage of mapped type
let stringPost: StringPost<Post> = {
   title: "How to do the wiggle",
   publishedDate: "15/05/2021",
   score: "2"
}

Taking a closer look at the mapped type definition:

type  StringPost<T> = {
   [K  in  keyof  T]: string
}

We can break down the code listed above as :

keyof T gets all properties of T
[K in keyof T] loop through all the properties of T
and finally [K in keyof T]: string, loop through all the properties of T and assign string as their type

See Mapped Types in Typescript for a more indepth exploration of Mapped Types.

Now that we are armed with the knowledge of these TypeScript features, let's now see how they are applied to create Utility types.

Demystifying Utility Types

Utility types are created from the application of some or all of the TypeScript features highlighted in previous sections. We will now take a look at the source code of the two utility types described above: Required<T> and Exclude<T, U> and see how they are defined.

The source code can be found in es5.d.ts

Required<T> Under The Hood

The Required<T> is defined as follows:

/**
* Make all properties in T required
*/
type Required<T> = {
    [P in keyof T]-?: T[P];
};

source

Let us break it apart:

This is a Mapped Type
keyof T gets all properties of T
[P in keyof T] loop through all the properties of T
the -? in [P in keyof T]-? means remove the character ? from the definition of the properties. This is how the optionality of the properties are removed and made required
The T[P] in the type definition means, take as type whatever property P is as part of type T. This way the original type of P is used in the mapped type.
Finally [P in keyof T]-?: T[P]; can be read as loop through all the properties of T remove the optional modifier ?, and keep original type.

That is it. Note that in the case where modifiers are to be added, the - can be dropped which defaults to adding, or +, can be used.

For example, instead of making all properties required, if we want to make them option, we would have the definition as [P in keyof T]+?: T[P]; or [P in keyof T]?: T[P];. In fact there is utility type for this and it is called Partial<T>.

Exclude<T, U> Under The Hood

The Exclude<T, U> is defined as follows:

/**
* Exclude from T those types that are assignable to U
*/
type Exclude<T, U> = T extends U ? never : T;

source

Let us break it apart:

This is a conditional type
Exclude<T, U> defines a generic with two variables: T and U
T extends U is the conditional check of the conditional type.
If T extends U then never is returned
If T extends U is false, then T is returned.

The T extends U is checking if T is a subtype of U. That is, if all properties of U can be found in T. If this not the case, then T is returned which would then be all the properties ofU with the properties found in T excluded.

This is how the exclusion in Exclude<T, U> is implemented.

As can be seen, even though utility types are powerful, there is nothing mysterious about them and they are understandable. They are created by using other features of TypeScript.

TypeScript comes with more utility types defined which can be found here, and hopefully this post now helps in appreciating what they are and how they are implemented to do what they do.

First impressions with Deno from building PlanetTypeScript.com

noreply@blogger.com (dade) — Mon, 07 Jun 2021 17:15:00 +0000

PlanetTypeScript.com is a content aggregator for all things TypeScript. It regularly polls RSS feeds for TypeScript related content, which it then list on the site, send out as a tweet via @planetypescript, and also as a weekly digest to subscribers.

Back when I used to write a lot of Java-related content on this blog, I found out that my blog ended up on www.topjavablogs.com/, which is a site that aggregates Java contents. So when I decided to play around with Deno, I thought a nice little idea would be to build something similar to topjavablogs.com, but for TypeScript. And that is how PlanetTypescript.com was born.

In this post, I will share some of my first impressions with Deno. The things I enjoyed and the bits I found inconvenient. Let's start with the cool stuff.

Native TypeScript Support

I find it super convenient that I could just write TypeScript without worrying about setting up a build chain that would take care of the transpilation to JavaScript. This is really nice and it makes TypeScript feel more like a standalone language in its own rights. This has to be one of the top-selling points of Deno for me.

Built-in utilities

The deno CLI comes with a whole lot of useful utilities and just comes in handy. For example deno info to check the dependencies. deno fmt, a code formatter, deno bundle for bundling modules and dependencies into a single file, deno compile for creating a self-contained executable, etc.

I personally find dealing with build pipelines within the JavaScript ecosystem to be a tiring chore. Having something like deno bundle or deno compile, built-in is super useful, as it takes care of most of the heavy lifting when it comes to deployment.

Web API

Deno seeks to provide implementations of Web API, and this I found to be really handy. For PlanetTypeScript.com, I need to make HTTP requests and also run background tasks. I could use fetch and web workers for this. API's I am already familiar with coming from the Web, and no need to install third-party libraries. This is super convenient.

Good Standard Library

In the course of building PlanetTypeScript.com, I needed to have logging, base64 encoding, use hash functions, and generating UUIDs. I was able to find modules within the Deno standard library that I could use for all of these. The Deno standard library is quite decent and has a battery-included feel to it. You can peruse the available modules here.

IDE Support

I primarily use Jetbrains IDEs. When not within a full-blown IDE environment, I use Microsoft Visual Studio Code. Both of these tools have plugins that implement support for Deno and they work very well. I was able to use the visual debugger in IntelliJ, which was quite handy.

So these are some of the cool things that made a good first impression on me with Deno. Below are some of the things that were a bit of a hassle.

No Registry and Package Manager.

Deno does not have any well-known central registry from which a package manager manages dependencies. Instead, Deno works with plain URLs. This setup leads to certain consequences which I believe hamper the developer experience.

For example, there is no central file to look into, to see all the dependencies of a project. To work around this, the convention in the Deno community is to re-import all dependencies from a file called deps.ts - basically recreating some sort of package.json.

Also, there is no easy way to quickly see if there are new versions of any of my project's dependencies. Or even better still, no command I could run from the command line that updates dependencies.

I later learned about Trex and dmm which could be used to achieve some of these tasks. I can't comment on them now as I have not gotten around to try them out yet.

From the perspective of the developer, I am yet to see the advantage the Deno model brings to the table. Instead, it feels to me more like a philosophical design decision, rather than a pragmatic one.

Nascent Library Ecosystem.

The inability to find needed third-party libraries proved to be one of the main inconveniences.

I had issues finding a third-party library to deal with OAuth. I tried Dashport, but it has become incompatible with recent versions of Deno and Oak (the HTTP middleware library I used). I tried using Firebase-auth, but there is no support for Deno.

I use MailChimp for sending the Weekly digests. Since the content of the emails would be different weekly, I needed to build the message dynamically. This involves interacting with MailChimp via its API. Unfortunately, MailChimp does not have an API client library for Deno, so I ended up having to directly call the needed HTTP endpoints.

Deno does have a compatibility layer that makes it possible to use Node/npm modules from within Deno. There are also CDN's like Skypack that make it possible to import npm modules for use within Deno. What I quickly found out, is that these workarounds, work best for the simplest of libraries. It does not work for anything slightly complicated. For example, I could not use any of these methods to get the Node library for Firebase-auth nor Mailchimp client library to work.

This nascency is strictly not a structural issue with Deno and something to be expected, given how new Deno is. This is also the sort of thing that would improve with time.

Conclusion.

I enjoyed using Deno. Having native TypeScript support, various CLI utilities, and a battery included-standard library, makes it easy to get productive real quick. The main inconveniences I encountered can be chucked up to Deno still being relatively new, but even with that, it is still a hell of a lot usable/productive for a 3 years old project.

3 Ways to Name Types in Typescript

noreply@blogger.com (dade) — Fri, 28 May 2021 05:59:00 +0000

TypeScript type's system is structural and hence, it allows for the easy creation of types based on the shape of data. These types can be created and used without having a name. For example:

let john: {name: string, age: number} = { 
  name: "John Doe", 
  age: 30
}

Where {name: string, age: number} is the type defined.

But there is a problem with using the type definition this way. If we want to create another person and annotate another variable with the type, we would need to repeat ourselves:

let john: {name: string, age: number} = {
  name: "John Doe", 
  age: 30
}

let jane: {name: string, age: number} = {
   name: "Jane Doe", 
   age: 45
}

This is not dry.

It is not dry because we have to repeat the types. A more efficient approach would be to define the type once, give it a name and then use the name when applying the type annotation. Fortunately, we can do exactly this.

TypeScript provides mechanisms for giving a name to a type, which can then be used when applying type annotations. We are going to look into the 3 different mechanisms through which this can be done. Namely: via classes, type aliases and interfaces.

Classes

A class can be seen as a blueprint for the creation of objects. For example given the following class definition:

class Person {
    name: string;
    age: number;

    constructor(name: string, age: number) {
        this.name = name;
        this.age = age;
      }

}

We can use it to create various instance objects:

// class definition
class Person {
    name: string;
    age: number;

    constructor(name: string, age: number) {
        this.name = name;
        this.age = age;
      }

}

// creating instances
let john = new Person("John Doe", 30)
let jane = new Person("Jane Doe", 45)

The interesting thing with classes is they are also a mechanism for defining and naming types. They not only serve as a blueprint for object creation, but they also create a type, which is named.

The Person class we defined also simultaneously creates a Person type, which we can use as a type annotation:

// class definition
class Person {
    name: string;
    age: number;

    constructor(name: string, age: number) {
        this.name = name;
        this.age = age;
      }

}

// annotating instances
let john: Person = new Person("John Doe", 30)
let jane: Person = new Person("Jane Doe", 45)

Type Aliase

Type aliases are another mechanism for naming types in TypeScript. It is a feature that makes it possible to give a name (or alias) to another type.

type Person = {name: string, age: number}
type MyString = string
// An alias can be given for an existing alias
type OhMyString = MyString

Hence it is best to see type aliases as a way to give a name to an already existing anonymous type, primitive type, or even another type alias.

Interfaces

Interfaces are another mechanism for creating types and naming them in TypeScript. It takes the following form:

interface Person {
    name: string,
    age: number
}

As can be seen, even though interface looks similar to using type aliases, they are not the same. Unlike type aliases, interfaces actually create a new type, and gives it a name, while type aliases only create a name for an existing type.

The 3 mechanisms shown thus far (classes, type aliases, and interfaces) provide us with 3 ways for doing the same thing: the naming of types. Even though they allow us to achieve similar things, there are some differences. In the next section, we take a look at some of these differences.

Difference Between Classes, Type Aliases, and Interfaces

The Presence or absence of a constructor

Classes differ from type aliases and interfaces in that it also provides a constructor that can be used to create well form instances of values for the type being defined. This is achieved by using the familiar new keyword:

// class definition
class Person {
    name: string;
    age: number;

    constructor(name: string, age: number) {
        this.name = name;
        this.age = age;
      }

}

// creating instances using constructor
let john: Person = new Person("John Doe", 30)

Constructors are not created as part of using type aliases nor interfaces.

Extension mechanism

Classes, type aliases, and interfaces all provide a mechanism for type extension, which is the ability to create a new type based on an already existing type. The only difference is the syntax used.

Extending types via classes takes the following form:

//type definition 
class Person {
    name: string;
    age: number;

    constructor(name: string, age: number) {
        this.name = name;
        this.age = age;
    }
      
}

// type extension
class Employee extends Person {
       isManager: boolean;
       
       constructor(name: string, age: number, isManager: boolean) {
           super(name, age)
           this.isManager = isManager
       }

}

doing the same via type aliases look like the following:

// type definition
type Person = {
    name: string
    age: number
}

// type extension
type Employee = Person & {isManager: boolean}

and finally with interface we have:

interface Person {
    name: string
    age: number
}

interface Employee extends Person {
    isManager: boolean
}

The syntax for extension with classes and interfaces looks similar while it is different for type aliases. Extending type aliases makes use of intersection types, which is a topic that is explored in details in Union and Intersection Types in TypeScript

Possibility for multiple definition

Interfaces differ from the other mechanisms in that, the type being named can be spread across multiple definitions.

//first definition 
interface Employee {
    name: string
}

// second definition
interface Employee {
    age: number
}

// third definition
interface Employee {
    isManager: boolean
}

// Merged Employee type being used
let john: Employee = {
    name: "John Doe",
    age: 30,
    isManager: true
}

TypeScript will take all the definitions of the interface Employee, and merges them into one. This is not possible with type aliases nor interfaces as their definition needs to be unique.

How to implement sleep function in JavaScript

noreply@blogger.com (dade) — Fri, 21 May 2021 23:06:00 +0000

Most popular mainstream languages have a sleep functionality that allows for the pausing of code execution for a specified amount of time. Not surprising this functionality is often implemented via a function usually named sleep. For example:

// ruby
sleep(time_secs)
// python
import time
time.sleep(time_secs)
// Java
Thread.sleep(time_milli_secs)
// Perl
sleep(time_secs)

JavaScript does not have a similar sleep function, although it has the setTimeout function which can be used to achieve a similar purpose. The only problem is, using setTimeout is not as convenient as one would like. This is due to the fact that setTimeout is an asynchronous function, hence one would have to continue the execution of the code, in the callback passed to it, after the elapsed time.

console.log("Executing code..."); 
setTimeout(() => { 
console.log("continue after pause"); 
}, 3000)

This is not convenient.

A more convenient option would be to have something similar to:

console.log("Executing code..."); 
// sleep for some time
// sleep()
console.log("continue after pause");

Unfortunately, this does not exist natively in JavaScript. But this is not to say we can't implement one!

We can implement such a sleep function by combining the capabilities of Promises and Async/Await in JavaScript. For a quick recap of the syntax involved, see Callback, Promise and Async/Await by Example in JavaScript.

The implementation of such a sleep function, involves "Promisifying" the setTimeout function, and then using the Async/Await syntax to provide a synchronous feel to a rather asynchronous function. It looks this:

function sleep(milliseconds) {
    return new Promise(resolve => {
        setTimeout(() => {
            resolve({})
        }, milliseconds);
    })
}

In a less verbose syntax this would be:

(milliseconds: number) => new Promise(resolve => setTimeout(resolve, milliseconds))

Usage would have to be within a function marked as async:

// definition
function sleep(milliseconds) {
    return new Promise(resolve => {
        setTimeout(() => {
            resolve({})
        }, milliseconds);
    })
}

//usage
async function doStuff() {
    console.log("start")
    await sleep(3000)
    console.log("continue")
}

doStuff()

The requirement to put the code in an async function is because, as of now, browsers and NodeJS do not support top-level async functions. Attempting to have the code used in the browser, outside of an async function, would lead to an error similar to this:

Uncaught SyntaxError: await is only valid in async functions and the top level bodies of modules

Attempting to run it in Node, would also lead to an error:

SyntaxError: await is only valid in async functions and the top level bodies of modules

Deno on the other hand, already supports top-level async functions, hence with Deno, the code can be used right away:

// definition
function sleep(milliseconds) {
    return new Promise(resolve => {
        setTimeout(() => {
            resolve({})
        }, milliseconds);
    })
}

//usage
console.log("start")
await sleep(3000)
console.log("continue")

Assuming the above code is in a file named script.js it can be executed using Deno as follows:

$> deno run script.js 
start
// pause for 3 seconds
continue

And there you have it: an implementation of a sleep function in JavaScript, which has a synchronous feel to it's usage. Definitely more convenient than using setTimeout directly.

Callback, Promise and Async/Await by Example in JavaScript

noreply@blogger.com (dade) — Thu, 20 May 2021 19:23:00 +0000

This post is going to show, by way of code examples, how to take a callback based API, modify it to use Promises and then use the Async/Await syntax. This post won't go into a detailed explanation of callbacks, promises or the Async/Await syntax. For such a detailed explanation of these concepts please check Asynchronous JavaScript, which is a section of MDN Web Docs, that explains asynchronicity and how callbacks, promises, and the Async/Await syntax helps with working with asynchronous JavaScript.

This post is meant for the developer who has somewhat of an understanding of asynchronicity in JavaScript, but requires a straight to the point code example to serve as a quick syntax reference for how to take a callback based API, update it to use promises and finally use the Async/Await with it.

For demonstration purposes, we are going to use the fs.readFile, which is a callback based API from reading files. We will have a file test.txt that would contain some text, we will then have a file script.js that would open the file, read the contents, and print it to the terminal.

The code will first be implemented using callbacks, it would then be updated to use Promises, and finally, instead of using Promise directly, it will be updated to use Async/Await.

Let's get started.

Using Callbacks

We first create a directory where we are going to work from, create also the file that will contain our code, and the two files that we would be reading from.

We first create the two files with contents.

$ mkdir ~/code
$ touch ~/code/script.js
$ echo "Beam me up, Scotty" > ~/code/test.txt
$ cd ~/code/

Next in the script.js file, we have the following code:

const fs = require("fs")

function readFileCallBack() {

fs.readFile("./test.txt", 'utf8',  (err, data) => {
  if (err) {
    console.error(err)
    return
  }
  
  console.log(data.trim() + " [callback]")
 })

}

readFileCallBack()

Executing the script by running node script.js should get "Beam me up, Scotty" printed to the terminal:

$ node script.js
Beam me up, Scotty [callback]

With the callback style, the result of the asynchronous operation is handled by a function passed in to the function performing the asynchronous operation.

Using Promises

Update script.js and add a version of readFileCallback that uses promises. It looks like this:

function readFilePromise() {
  return new Promise((resolve, reject) => {
     fs.readFile("./test.txt", 'utf8',  (err, data) => {
     if (err) {
       reject(err)
       return
     }

      resolve(data.trim())
    })
  });
}


readFilePromise()
 .then(data => console.log(data  + " [promise]"))
 .catch(err => console.log(err))

Execute the script by running node script.js:

$ node script.js
Beam me up, Scotty [callback]
Beam me up, Scotty [promise]

With promises, the result of the asynchronous operation is handled by a function passed to the then function exposed by the promise object.

Using Async/Await

Update script.js and add a third version that uses the Async/Await syntax. Since Async/Await is a syntax that makes using promises easier, the Async/Await implementation would make use of the readFilePromise() function. It looks like this:

async function readFileAsync() {
  try {
    const data = await readFilePromise()
    console.log(data.trim() + " [async-await]")
  } catch (err) {
    console.log(err)
  }
}

readFileAsync()

Executing the script by running node script.js will print something similar to this, to the terminal:

Beam me up, Scotty [callback]
Beam me up, Scotty [promise]
Beam me up, Scotty [async-await]

With async/await, the result of the asynchronous operation is handled as if it was a synchronous operation. The await is responsible for this, while the function in which it is used has to be preceeded with async keyword.

The complete file with the 3 implementations is presented below:

const fs = require("fs")

// callback
function readFileCallBack() {

fs.readFile("./test.txt", 'utf8',  (err, data) => {
  if (err) {
    console.error(err)
    return
  }
  console.log(data.trim() + " [callback]")
  
 })

}

readFileCallBack()

// promise
function readFilePromise() {
  return new Promise((resolve, reject) => {
     fs.readFile("./test.txt", 'utf8',  (err, data) => {
     if (err) {
       reject(err)
       return
     }

      resolve(data.trim())
    })
  });
}


readFilePromise()
 .then(data => console.log(data  + " [promise]"))
 .catch(err => console.log(err))


// async/await
async function readFileAsync() {
  try {
    const data = await readFilePromise()
    console.log(data.trim() + " [async-await]")
  } catch (err) {
    console.log(err)
  }
}

readFileAsync()

Error Handling

To illustrate that the error handling in the 3 implementation work as expected, rename the test.txt file and rerun the script:


$ mv test.txt test.txt.backup
$ node script.js
[Error: ENOENT: no such file or directory, open './test.txt'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: './test.txt'
}
[Error: ENOENT: no such file or directory, open './test.txt'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: './test.txt'
}
[Error: ENOENT: no such file or directory, open './test.txt'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: './test.txt'
}

Showing that the error handling code, which is to just print the error to the console, works as expected in the 3 implementations.

ip-num v1.3.2 has been released

noreply@blogger.com (dade) — Sun, 09 May 2021 07:51:00 +0000

ip-num is A TypeScript/JavaScript library for working with ASN, IPv4, and IPv6 numbers. It provides representations of these internet protocol numbers with the ability to perform various IP related operations like parsing, validating etc, on them.

A new version of ip-num, version 1.3.2 is now available.

This release contains one new feature and a bug fix.

Add method to split range into smaller ranges of certain size

In previous versions of ip-num, it was possible to take IP ranges with prefix less than /32 and split them into two. For example:

import {IPv4CidrRange} from "ip-num/IPRange";
import {IPv4Prefix} from "ip-num/Prefix";

let ipv4CidrRange = IPv4CidrRange.fromCidr("192.168.208.0/24");
let splitRanges: Array<IPv4CidrRange> = ipv4CidrRange.split();

// console logs:
// 192.168.208.0/25
// 192.168.208.128/25

splitRanges.forEach(range => console.log(range.toCidrString()))

But what if one needs to take an IP range and instead of just splitting into two, there is the requirement to split into a number of ranges with a specified prefix? Well, that is now possible with the addition of splitInto method:

import {IPv4CidrRange} from "ip-num/IPRange";
import {IPv4Prefix} from "ip-num/Prefix";

let ipv4CidrRange = IPv4CidrRange.fromCidr("192.168.208.0/24");
let splitRanges: Array<IPv4CidrRange> = 
ipv4CidrRange.splitInto(IPv4Prefix.fromNumber(26));

// console logs:
// 192.168.208.0/26
// 192.168.208.64/26
// 192.168.208.128/26
// 192.168.208.192/26

splitRanges.forEach(range => console.log(range.toCidrString()))

RangedSet.isCidrAble() incorrect results

Previous versions of ip-num failed to properly report that a string representing a single IP can be represented in the CIDR notation. This has now been fixed:

import {RangedSet} from "ip-num/IPRange";

let rangedSet= RangedSet.fromRangeString("1.2.3.4-1.2.3.4");

// now returns true
console.log(rangedSet.isCidrAble())

As always, ip-num is just an npm install or npm upgrade away.

Feel free to open an issue to discuss a feature or to report a bug.

Mapped Types in Typescript

noreply@blogger.com (dade) — Sat, 24 Apr 2021 23:21:00 +0000

This post will explore Mapped Types, another advanced feature of the Typescript type system. It is part of the Introduction to Advanced Types in TypeScript series.

A useful mental model to have when approaching some of the advanced type system features of Typescript is to view them as a mechanism for constructing other types from existing types. This view is spot on when it comes to mapped types as they are a mechanism that Typescript provides by constructing new types by mapping existing types into new ones.

This post would show how this looks like. It would be as beginner-friendly as possible, but having some knowledge of Generics, Union Types and Literal types in Typescript would be a plus.

To kickstart we take a quick again, at the keyof Operator, as it is essential to the workings of Mapped Types.

keyof Operator

The key idea of Mapped Types is to create a new type by mapping (read modifying) the properties of existing types. And how do we get access to these properties? We use the keyof operator. As explained in Introduction to Index Types in Typescript, The keyof type operator is an operator that allows the retrieval of all the indexes (read properties) of a type as a union type. The individual components of the retrieved union types are the literal types that represent the properties of that type.

This can be illustrated as follows. Given the following type definition:

type Employee = {
name: string
employedOn: Date
age: string
role: {
  title: string
  salary: number
  isManagement: boolean
 }
}

The properties can be extracted as a union type as follows:

type Properties = keyof Employee

We can demonstrate that Properties is indeed a union type that consists of literal types based off the properties of Employee:

var value: Properties = "name" // compiles
var value: Properties = "employedOn" // compiles
var value: Properties = "age" // compiles
var value: Properties = "role" // compiles
var value: Properties = "gender" // does not compiles

All the variable definition compiles except the one where "gender" is to be assigned to a variable of Properties. And the reason why this does not compile is due to the fact that "gender" is not part of the properties of the Employee type.

So in summary, the keyof operator allows us to get access to the properties of a type as a union type.

A Mapped Type

A mapped type is then a type that is created by accessing the properties of another type, using the keyof operator, and modifying the extracted properties in some sort of way to create another type. As an example, given a type definition as follows:

type Person = {
    age: string | number
    height: string | number
}

This type can be turned into another type where the properties are mapped from type string | number to string. To achieve that, a mapped type can be defined as such:

type ToStringProps<T> = {
    [P in keyof T]: string
}

Which can then be used on the Person type as follows:

type StringPerson = ToStringProps<Person>

var person: StringPerson = {
    age: "10",
    height: "6"
}

The StringPerson type thus created can only have properties of type string.

The Mapped type above is ToStringProps<T> and in this case, it is used to modify the return type. But it is not only the return type of properties that can be modified using Mapped Types, property names and property identifiers (mutability and optionality identifier) can also be modified. The next example shows how these looks.

Modifying Identifiers using Mapped Types

Two identifiers can be modified using Mapped types. These are mutability, which is implemented via the readonly keyword, and optionality which is implemented via the ? symbol. The modification that can be achieved via mapped types involves adding or removing these identifiers. To add the identifier the Mapped type definition makes use of the + symbol, while to remove the target identifier, the - symbol is used.

To illustrate, a Mapped type that takes a type and produced an immutable version of it would look like this:

type Immutable<T> = {
    +readonly [P in keyof T]: T[P]
}

While a mapped type that takes a type and produced a mutable version would look like this:

type Mmutable<T> = {
    -readonly [P in keyof T]: T[P]
}

Note, when no symbol is used, then addition is assumed. Hence the Immutable mapped type can also be defined like this:

type Immutable<T> = {
    readonly [P in keyof T]: T[P]
}

The above examples show how the readonly which is responsible for mutability can be mapped. On the other hand, to illustrate how a mapped type can be used to produce another type with all of its properties turned optional, we have:

type Optional<T> = {
    [P in keyof T]+? : T[P]
}

While a mapped type that takes a type and produced another type where all its properties would have to be specified will look like this:

type NonOptional<T> = {
    [P in keyof T]-? : T[P]
}

Again, note, when no symbol is used, then addition is assumed. Hence the Optional mapped type can also be defined like this:

type Optional<T> = {
   [P in keyof T]? : T[P]
}

Modifying Property Name using Mapped Types

Mapped type can also be used to produce a new type by modifying the property names of an existing type. This is achieved by using the as keyword, in combination with Template literals. For example, a mapped type that updates the given type by prepending "_" to the property names will look like this:

type UnderlineProp<T> = {
    [P in keyof T as `_${string & P}`]: T[P]
}

which can then be used as follows:

type Person = {
    name: string
    age: number
}

type _Person = UnderlineProp<Person>

var _person: _Person = {
    _name: "John",
    _age: 20
}

In the above example _Person is a type produced from Person type, using the mapped type definition of UnderlineProp.

Mapped types are quite powerful and allow for sophisticated manipulation of types in Typescript, especially when used with other features of the type system. Above, it is used together with Template literals. It is also possible to make use mapped types together with Conditional types in determining the type of property to be updated.

This post is part of a series. See Introduction to Advance Types in TypeScript for more posts in the series