Creating a custom Temporal data convertor for encryption

Background: Nightfall services & discovery of Temporal

Nightfall is a data security vendor that integrates with our customers’ third party applications (Slack, Google Drive, Github, Jira etc) to, on a continuous real-time basis, scan all content being added to these applications for sensitive data. 

As an example use case, many of our fintech customers will leverage our platform to continuously scan slack files and messages for sensitive banking information (credit card numbers, banking numbers, etc.) to ensure that they are PCI-DSS compliant and continuously maintain good data security hygiene. Given the high stakes nature of our products, durability, high availability, and fault-tolerance are commonly required when building services at Nightfall. 

In the above customer use case, we need to ensure that we scan every single Slack file and message as promised. Thus in an effort to ensure we were appropriately dealing with failure, when we built the first version of our Slack integration we designed our own stateful workflow orchestration system to meet these system requirements. Soon after this, Temporal had come out of stealth and announced its fundraising round and 1.0 release. Due to the complexity of building and maintaining an effective stateful workflow orchestration system, we decided to see if Temporal could meet our needs for the next integration that we would build. 

Read here more information about stateful workflow orchestration and how Temporal works.

How Temporal handles payload encryption

As a security vendor we have strict rules around making sure traffic to any data stores are encrypted in transit and at rest. By default, Temporal payloads are stored unencrypted inside of its data store. The consequence of this is that these payloads can be read from the temporal UI and from the CLI (Command Line Interface) as plaintext — see the screenshot below.

How Temporal payloads work

The payloads that Temporal stores are mostly function arguments and function results that are converted/marshaled data into a byte array: Temporal Payload Struct Example

For example, let's say we have defined a Temporal Activity called “GetSlackFileMetaDataActivity” which takes a company UUID and fileID as parameters. This Activity will eventually return an integer as part of its response payload when the Activity is executed and completed.

What this function/Activity would look like in service code written in Golang:

Let's say we supply the function with the following parameters “16f336dc-f3dc-48c9-99d6-502582d009fd” and “abc123”. If we look in the temporal UI after the function has been queued and executed it will look something like this:

The “input” array are function arguments. The “result” is the function result.

Finding a solution for sensitive data in Temporal payloads

Does this mean we can’t use Temporal if we need to work with sensitive data? The simple answer is “no”. Luckily Temporal is extremely flexible and configurable down to the client/worker/workflow/activity levels. The client construction level we learned may pass in a custom data converter in client options which overrides default payload converter that just does marshalling/unmarshalling data. A custom data converter allows us to encrypt payloads and send them over to the Temporal server, then to be saved in the database to be retrieved later when the activity or workflow needs to be run. What is even more interesting is that the Temporal server is agnostic to the data and doesn’t need to decrypt it to allow it to work. Hence the encryption and decryption happens inside your microservice and your cryptographical secret is never shared with Temporal.

Now let's try our GetSlackFileMetaDataActivity again with our custom Temporal payload encryption converter turned on:

If you wish not to see encrypted payloads when loading the data to the UI and still yield the benefit of encryption, Temporal has the ability to pass in a custom data converter into the UI using Temporal CLI, allowing you to see decrypted payloads. This is similar to what you would see in the decrypted version of the UI:

Implementing a Custom Data Converter - Part 1

Before implementing our own custom data converter we first needed to know how the original default data converter worked (here). If you look at the interface you will see the data converter interface functions are converting to and from payloads. The second step was copying over the NewCompositeConverter (here) so we can modify it to add our encryption. This will help us make our own data converter to match this interface.

Figuring out encryption

Next, we needed to figure out an encryption standard best suited for our needs. We chose Advanced Encryption Standard (AES) to encrypt our customer’s sensitive data, as AES is an industry standard and sees wide application use from WiFi to NSA secret documents. Also in Golang, AES is highly performant. In fact Golang AES documentation (here) indicates that encryption can perform at constant time with hardware acceleration if your CPU supports it.

Implementing a Custom Data Converter - Part 2

Now that we have decided our encryption standard, it is now time to build the encryption functions Encrypt/Decrypt, which operate on byte arrays of data.

Finally we add our encryption functions on “ToPayload()” and denote it was encrypted in the metadata. Then on “FromPayload()”, we check if it was encrypted in metadata and then decrypt. After this, we are done and we just need to add the data converter to the temporal client options or use “workflow.WithDataConverter()” on specific workflows. 

Implementing encryption in our Custom Data Converter

Implementing compression in our Custom Data Converter

Temporal supports compression in their default payload converter which allows one to squeeze a bit more out of Temporal history. However, given the highly sensitive nature of our business we decided to remove compression due to possible BREACH and CRIME attacks. You can find our Golang Data Converter Encryption Code on GitHub.

This is how we have ensured all sensitive data stays encrypted in our workflows, by using the above custom data converter to encrypt payloads in Temporal. We will continue to share more learnings, solutions, and approaches through our engineering blogs. If you are interested in working on some of what we have discussed above or in joining us to build the first cloud-native data loss prevention (DLP) platform — we are hiring across our teams! We'd love to have you join us.