Multi-Dimensional Microscopy Datasets: Storing, Processing, and Visualizing • iBiology

00:00:07.19 Hi, my name is Loic Royer.
00:00:09.19 I am a group leader at The Chan Zuckerberg Biohub.
00:00:11.28 Today, I will lay out the basics on how to store, process, and visualize
00:00:16.12 N-dimensional microscopy datasets.
00:00:18.12 This presentation is part of the image analysis course.
00:00:22.21 So, microscopy starts at the microscope,
00:00:25.14 by acquiring the image data on a camera or detector.
00:00:28.16 This data is then stored and processed on the computer.
00:00:31.12 And finally, the data can be visualized on the screen.
00:00:34.23 Until recently, most microscopy data
00:00:37.22 -- as you can see here --
00:00:39.11 was 2-dimensional or had limited depth.
00:00:43.01 Nowadays, more and more microscope designs and modalities
00:00:46.08 produce 3-dimensional datasets
00:00:48.04 that are extending across time and other dimensions,
00:00:51.07 thus producing multi-dimensional data.
00:00:54.22 A 3-dimensional image is often called a stack.
00:00:57.25 Images with more dimensions are also often called stacks.
00:01:01.18 A stack is made of voxels in the same way
00:01:04.07 that a 2-dimensional image is made of pixels.
00:01:06.29 Each voxel is assigned a fractional or integer value.
00:01:09.26 These values can be represented with different levels of precision,
00:01:13.13 such as 8, 16, 32, or 64 bits.
00:01:16.20 But typically, we don't just acquire a single image.
00:01:20.10 In the case of live imaging,
00:01:21.29 you typically require time lapses consisting of
00:01:25.01 multiple stacks obtained at regular intervals.
00:01:27.26 You can also acquire multiple channels
00:01:30.04 that may represent different imaging wavelengths
00:01:32.02 or imaging modalities, such as different fluorescent labels,
00:01:36.20 different label-free modalities, etc.
00:01:40.16 Throughout this presentation,
00:01:42.04 I will use a running example,
00:01:43.25 a 21-hour time lapse recording of a developing drosophila embryo.
00:01:48.26 What you see here are the histones labeled with red fluorescent protein.
00:01:53.04 The acquisition was done on a light sheet microscope
00:01:55.26 with 30-second intervals
00:01:59.15 for a total of 2500 time points.
00:02:02.25 The raw data, before processing, is a total of 2 terabytes.
00:02:07.15 Such datasets pose a number of challenges
00:02:09.24 in terms of storage, processing, and visualization.
00:02:14.05 Some of these challenges are specific
00:02:16.09 to their multi-dimensional nature.
00:02:18.17 At the end of this lecture,
00:02:20.05 you will have a better understanding of
00:02:22.21 how to store, process, and visualize such datasets.
00:02:26.03 Attached to this video,
00:02:28.07 we will provide all references and links to the tools mentioned
00:02:31.17 and used in this presentation,
00:02:33.28 as well as alternatives and complementary software.
00:02:38.13 First, let's examine the key concepts and challenges
00:02:41.25 that arise when storing such data.
00:02:46.08 In this plot, we have the storage capacity on the y axis
00:02:49.25 and storage speed on the x axis
00:02:52.12 for different acquisition or storage devices.
00:02:55.26 For example, you can see that hard drives
00:02:58.26 can typically store 2 terabytes
00:03:01.13 at a speed of 100 megabytes per second.
00:03:03.29 The Drosophila time lapse that I showed you before
00:03:07.13 is 2 terabytes of raw data
00:03:09.11 and is required on a microscope that can output
00:03:11.19 2 gigabytes per second.
00:03:13.20 Faster microscopes exist that can out...
00:03:16.15 output up to 4 gigabytes per second
00:03:18.29 and produce up to 50 terabytes of data on a single run.
00:03:23.08 The take-home message here is that
00:03:27.01 modern microscopes produce very large amounts of data
00:03:29.07 that pose serious data management challenges.
00:03:32.13 Let's first review some basic concepts on how to store images.
00:03:38.01 When storing such datasets,
00:03:40.11 an important consideration is how to decide on the unit of storage.
00:03:44.00 For example, each time point is typically stored as one unit
00:03:47.24 that is materialized as a single file.
00:03:50.18 However, if you attempt to access the value of a single voxel across time,
00:03:55.10 you are often forced to load the entire stack of each time point.
00:04:00.02 This is highly inefficient.
00:04:02.07 To address this, a common strategy
00:04:04.25 consists of partitioning the dataset into smaller storage units
00:04:07.24 called chunks.
00:04:09.07 Ideally, the partitioning happens along all dimensions,
00:04:13.03 thus facilitating localized access.
00:04:17.00 Now you can access the individual voxels
00:04:19.13 by loading the relevant chunks
00:04:21.25 without having to load the entire stack.
00:04:24.19 Oftentimes, one needs to process or visualize the data
00:04:27.22 at a lower level of detail.
00:04:30.15 Therefore, it is inefficient to have to load every single voxel of a stack.
00:04:35.05 In order to have rapid access to coarser representations of the data,
00:04:39.07 it is often convenient to store
00:04:42.19 downscaled versions of the image along with the full resolution data.
00:04:46.20 Such representations are called image pyramids.
00:04:50.17 The additional data that you need to be...
00:04:54.02 to be stored is quite limited.
00:04:55.22 For example, if we store two lower-resolution versions of the data
00:04:59.16 by downscaling by a factor of 2 along every dimension,
00:05:02.20 only 14% extra data needs to be stored.
00:05:07.12 In some cases, it can be advantageous
00:05:10.05 to have chunks of different dimensions
00:05:13.14 within a single stack.
00:05:15.14 This affords the opportunity to adapt the storage
00:05:18.06 to the information content present in different image regions.
00:05:23.19 This can be useful when representing very large stacks
00:05:25.29 that are otherwise difficult to
00:05:28.05 store, process, or visualize.
00:05:35.01 Now, we are considering images
00:05:38.03 with, potentially, several dimensions.
00:05:40.15 But at the end, most storage devices
00:05:42.19 -- such as computer memory or hard drives --
00:05:44.24 can only hold data along a single dimension.
00:05:48.07 Therefore, we need a strategy to
00:05:51.08 linearize a multi-dimensional data set
00:05:53.18 into a single sequence of numbers.
00:05:56.09 A typical approach is to first list the values
00:05:59.00 along the X dimension and then along the Y dimension,
00:06:01.24 and so on for all dimensions.
00:06:04.28 In this case, voxel values that are close along the X and Y axes
00:06:08.16 are also in proximity when linearized.
00:06:11.13 For example, here, pixels 6 and 7
00:06:15.10 are neighbors along the X axis
00:06:17.07 and also neighbors after linearization.
00:06:20.02 Another possibility is to order the voxels
00:06:24.21 along first the Z axis and then the Y axis
00:06:27.27 and finally the X axis.
00:06:29.15 In that case, pixels 6 and 7
00:06:32.29 are also neighbors along the Z axis.
00:06:37.07 Proxim... proximity of voxels in computer memory
00:06:41.07 or on storage devices
00:06:43.15 has profound consequences and is critical for access speed
00:06:46.27 when writing, reading, and processing image data.
00:06:49.29 So, this is very important and really helps
00:06:51.25 accessing the data in an efficient manner.
00:06:54.10 Another way to mitigate the large amount of data
00:06:57.25 produced by modern microscopes
00:07:02.00 is data compression,
00:07:04.04 and this has become increasingly important.
00:07:06.11 Once chunks of the data have been linearized,
00:07:08.25 compression algorithm...
00:07:10.21 algorithms can be used to reduce their size.
00:07:13.20 Lossless compression algorithms such as ZIP
00:07:17.08 have the advantage that they do not destroy information
00:07:19.07 and are perfectly reversible.
00:07:21.03 However, they are limited,
00:07:23.19 and rarely achieve compression factors above 4 on image data.
00:07:26.12 Lossy compression algorithms
00:07:28.04 can achieve much higher compression ratios,
00:07:30.15 up to a factor of 100 at times,
00:07:32.16 but they must necessarily discard information
00:07:34.24 to achieve such high levels of compression.
00:07:37.22 Another important consideration is that of compression speed.
00:07:41.16 Some algorithms might achieve higher compression ratios
00:07:44.20 without discarding too much information,
00:07:47.08 but they may be prohibitively slow.
00:07:51.15 The computer that controls the microscope
00:07:54.06 rarely has the storage capacity to store the acquired data long-term.
00:07:57.12 Eventually, the data needs to be transferred
00:08:00.11 to local storage or to the cloud.
00:08:03.07 Cloud storage has the advantage
00:08:05.27 that it is often well-maintained,
00:08:07.13 comes with features such as backups and 24-hour support,
00:08:10.13 and is for all practical purposes
00:08:13.07 unlimited in capacity.
00:08:14.29 Yet, it has a cost that can be quite significant.
00:08:19.01 In contrast, local storage has the advantage of
00:08:22.23 easy access, usage flexibility,
00:08:24.29 and high-speed data transfers.
00:08:27.23 But it requires maintenance, expertise,
00:08:30.05 and it's necessarily limited in capacity.
00:08:33.03 So, we have seen how to store the data.
00:08:35.25 Now, the question is, how to process it?
00:08:38.06 And what are the challenges and solutions around
00:08:42.24 multi-dimensional image processing.
00:08:45.08 So, once acquired and stored,
00:08:46.26 the data typically needs to be processed.
00:08:50.02 This can be done on a single machine
00:08:52.26 or on a cluster of machines.
00:08:55.00 These machines can be local or in the cloud.
00:08:58.00 Transmission speed and ease of access of the data
00:09:00.26 are key considerations.
00:09:04.14 So, what are the typical processing steps
00:09:07.02 of these 3D, 4D, and multi-dimensional data sets?
00:09:11.27 So, first you have to denoise the data.
00:09:15.27 Then you have to correct for the background.
00:09:19.28 And then you have to deconvolve, typically, the data.
00:09:23.12 Other possible subsequent steps are, for example,
00:09:26.01 stitching and fusing the images together.
00:09:30.14 Some of these processing steps fundamentally require you to
00:09:33.23 take into account the multi-dimensional nature of the data.
00:09:36.25 For example, 3D deconvolution
00:09:39.09 typically requires knowing the 3D structure of the point spread function
00:09:41.28 to reduce blur and out of focus light.
00:09:46.10 Another issue in 3D volumetric imaging
00:09:48.10 is that of the anisotropic nature of acquired 3D images.
00:09:53.00 Most microscopes acquire the data plane by plane.
00:09:57.26 Unfortunately, it's not practical to have as many planes
00:10:02.00 as we have pixels on a single 2D image.
00:10:06.14 And so, while the logical layout of pixel
00:10:09.02 is as shown here,
00:10:11.05 the true shape of a voxel, in truth,
00:10:13.21 is elongated along the Z axis,
00:10:16.03 so it's more like a brick.
00:10:17.17 Recently, methods that utilize machine learning and deep learning,
00:10:20.14 in particular,
00:10:22.05 have been developed that can turn anisotropic acquisitions
00:10:24.06 into isotropic stacks.
00:10:26.10 As you can see here,
00:10:28.06 the resolution along the axial dimension
00:10:30.04 -- the Z axis --
00:10:31.26 is initially quite poor,
00:10:34.22 but can be improved, giving an isotropic stack.
00:10:39.02 In general, deep learning is becoming an attractive approach
00:10:42.15 for processing multi-dimensional datasets.
00:10:44.22 First, because the results are oftentimes impressive,
00:10:47.11 but also because deep learning leverages
00:10:49.23 specialized computing hardware
00:10:51.28 that makes these algorithms quite efficient.
00:10:54.27 Indeed, in all of these examples mentioned previously,
00:10:58.13 image processing and computation around a single voxel
00:11:01.29 requires us to consider a large number of neighboring voxels.
00:11:06.01 Already, when considering only adjacent voxels,
00:11:09.09 the more dimensions the more voxels need to be considered.
00:11:13.05 This leads to a rapid explosion of the computational requirements.
00:11:17.09 For example, in 4 dimensions,
00:11:19.12 a single voxel has 8 direct neighbors,
00:11:22.09 and a total of 18 neighbors when including diagonal neighbors.
00:11:25.27 And this is only for the most direct neighbors surrounding that voxel.
00:11:31.24 When in the past most image processing
00:11:34.16 was done on the central processing units of computers,
00:11:36.26 today the processing is shifting to graphical processing units.
00:11:41.23 These GPUs are designed to
00:11:44.06 operate on arrays of voxels in parallel
00:11:46.14 and are thus oftentimes
00:11:48.18 100 times faster than CPUs for image processing.
00:11:53.06 Okay.
00:11:54.13 So, we have seen how to store,
00:11:56.10 how to process these datasets.
00:11:58.07 Now, let's look at how to visualize this data.
00:12:03.01 So, one simple approach is to visualize on the screen
00:12:06.09 each 2D plane for different time points and channels.
00:12:09.13 Here we show a selection of 6 planes across a 3D stack
00:12:13.21 of our Drosophila embryo dataset.
00:12:16.08 We can also display these slices over time as a movie,
00:12:21.03 or project all the slices by taking the maximal value
00:12:27.03 for voxels along... along the Z direction.
00:12:30.00 However, a lot of details from the dorsal and ventral side of the embryo
00:12:33.22 get projected together into a single 2D image,
00:12:36.12 leading to confusion.
00:12:39.28 Instead, a better idea is to
00:12:42.12 project only half of the stack at a time,
00:12:46.00 into two distinct ventral and dorsal projections.
00:12:49.10 Another possibility is to project and color code the depth of the visible pixels
00:12:54.24 to convey depth explicitly.
00:12:56.20 However, color is oftentimes a poor conveyor of depth.
00:13:01.25 Yet another approach is to simultaneously and interactively
00:13:05.13 visualize three orthogonal slices
00:13:07.06 -- XY, XZ, and YZ --
00:13:09.27 and all of that... intersecting around a single point.
00:13:18.21 We have just seen very simple schemes
00:13:20.25 to visualize a 3-dimensional dataset into a 2D screen.
00:13:24.24 These schemes are unsatisfactory,
00:13:27.01 as they don't convey the true 3-dimensional nature of the data.
00:13:31.24 Creating 2D images from multi-dimensional data
00:13:34.24 is often called rendering.
00:13:36.18 There are mainly two classical approaches for rendering 3D data:
00:13:41.25 volume rendering and surface rendering, as you can see here.
00:13:47.00 In the following, I will focus on volume rendering,
00:13:50.16 as it is oftentimes the most appropriate
00:13:54.14 for light microscopy data.
00:13:57.08 A typical technique for volume rendering
00:14:00.09 is that of ray casting.
00:14:02.06 Rays are drawn from the eye to each pixel on the screen
00:14:06.15 and continue traversing the whole 3-dimensional array of voxels.
00:14:12.00 For each such ray, we collect the voxels' values along the ray,
00:14:16.13 and then decide on which intensity to display on the screen
00:14:19.27 for that pixel.
00:14:22.13 The typical approach simply collects the maximal value along the ray.
00:14:25.25 More sophisticated schemes
00:14:28.07 that model attenuation, color, and lighting
00:14:32.16 give more informative and pleasing results.
00:14:36.11 Here's, for example, what you get if you
00:14:40.21 collect the maximal voxel intensity along each ray.
00:14:43.16 The problem is that you also see what is at the back of the embryo,
00:14:47.22 making the rendering, again, confusing.
00:14:50.14 We need something better.
00:14:52.24 Let's consider an example.
00:14:55.05 Here, the maximal projection would return a value
00:15:00.16 coming from the back of the visualized sample,
00:15:03.12 despite the fact that there's a lot of structure
00:15:05.25 to show at the front of it.
00:15:08.02 One simple solution is to model the attenuation
00:15:10.17 as a prop...
00:15:12.10 as proportional to the signal itself.
00:15:14.22 Now, intensity values are attenuated
00:15:18.14 as they traverse the sample.
00:15:22.09 The maximal val... voxel value is not at the back of the sample anymore,
00:15:26.27 but at the front.
00:15:29.13 Because there's less attenuation for structures at the front
00:15:33.11 than at the back.
00:15:34.19 Here is the result of rendering with max projection and attenuation.
00:15:37.15 Structures on the surface are easier to discern.
00:15:45.13 We can also slice the 3D image to see the interior.
00:15:52.13 Time lapses can also be visualized
00:15:55.10 and combined with rotation.
00:15:56.21 Here you can see two views that are 180 degrees relative to each other,
00:16:00.29 with time lapse playback.
00:16:06.18 We can also combine all of these aspects together,
00:16:09.08 rotating and slicing the data set
00:16:12.01 while playing back the time lapse.
00:16:14.00 To be of practical use,
00:16:15.29 volume rendering needs to be fast and fluid,
00:16:18.28 especially during interactive visualization.
00:16:22.18 And so, we need some tricks to be able to do that fast.
00:16:26.00 So, here I'm gonna show you one.
00:16:28.05 So, the standard approach for volume rendering
00:16:31.09 is simply to go through every voxel on the array,
00:16:33.27 one by one,
00:16:36.17 and collect all the information that needs to be collected.
00:16:38.27 The problem is that it's oftentimes too slow
00:16:43.14 and not efficient.
00:16:44.28 A better approach is to have a multipass rendering algorithm,
00:16:48.03 where you do multiple passes collecting a fraction of the information,
00:16:51.19 and progressively you refine the image.
00:16:55.01 And so, the next thing to do is then to
00:16:59.22 maybe skip every...
00:17:01.16 to only collect information every 4 voxels, as shown here.
00:17:05.10 And you do it once, and then again and again and again.
00:17:07.24 And... and so, while you rotate your sample,
00:17:10.12 you only show 1/4 of the information,
00:17:13.07 in some sense.
00:17:15.13 And then, when you stop rotating,
00:17:17.14 stop interacting with your dataset,
00:17:19.06 you can then refine and then complete the image
00:17:21.22 to have the full image quality rendering, right?
00:17:24.12 So, that's the successive approach.
00:17:27.05 But we need something smarter.
00:17:31.20 Because the problem is that, if you look at it,
00:17:34.01 there's big holes in the sampling pattern for a long time.
00:17:35.24 So, a better strategy is something that we called Fibonacci rendering.
00:17:41.05 And the idea there is to
00:17:44.28 basically shift the pattern in an optimal way.
00:17:46.21 And we use the mathematical properties of the Fibonacci sequence
00:17:49.01 and the golden ratio to achieve that.
00:17:52.01 So, let's look at how it looks like.
00:17:56.07 So, instead of collecting the samples
00:17:59.06 next to the previous ones,
00:18:01.14 and thus skipping whole...
00:18:04.18 really big holes in the sampling pattern for a long time,
00:18:07.15 we simply jump ahead and try to collect information
00:18:11.02 from the middle point,
00:18:12.24 or a little bit before that or a bit beyond that,
00:18:15.22 and we repeat that every time
00:18:19.08 by somehow keeping the holes in the sampling pattern
00:18:21.09 as small as possible as soon as possible.
00:18:23.10 And so, this technique, as you can see here, is...
00:18:27.01 leads to essentially less a perception of blinking or flashing.
00:18:34.04 As you can see here, the naive approach is... is very slow,
00:18:37.22 so it takes a long time to get the image,
00:18:39.16 so it's not going to be interactive.
00:18:41.07 But in the case of the Fibbon... Fibonacci approach,
00:18:43.19 the image...
00:18:47.01 after a few passes, the image is already of quite good quality.
00:18:49.23 Here is how it looks like in practice.
00:18:52.17 This is a very large multi-channel image of neurons within the Drosophila brain.
00:18:57.05 As you can see, the interaction is fluid,
00:19:00.11 and the quality recovers as soon as the movement is stopped.
00:19:04.26 The multipass algorithm is active during interaction.
00:19:08.00 And when we stop interacting with the volume,
00:19:11.06 the image is refined and finally displayed at its full quality.
00:19:16.18 In conclusion, modern microscopy
00:19:20.00 produces large multi-dimensional data sets
00:19:22.07 that pose novel challenges for storage, processing, and visualization.
00:19:26.26 I hope that this presentation has given you
00:19:29.18 a basic understanding of the concepts and challenges
00:19:32.10 posed by modern multi-dimensional microscopy data.
00:19:37.07 Thank you very much.

This Talk

Speaker: Loic Royer

Audience:

Educators of Adv. Undergrad / Grad
Researcher
Educators

Recorded: June 2019

Talk Overview

Modern microscopy produces large multi-dimensional datasets, which creates new challenges for data storage, processing and visualization. In this talk, Dr. Loic Royer uses a developing drosophila melanogaster embryo to explain how to solve some of the challenges produced by multi-dimensional microscopy datasets.

Concepts: Large dataset, multi-dimensional dataset, store, process, and visualize large datasets, voxels, lossless and lossy compression algorithms, deconvolution, rendering, fibonacci rendering

Questions

What are the common techniques used to improve multi-dimensional image storage efficiency?
What are the typical image processing steps after image acquisition?
Why is max projection not always the best way to visualise a 3D light microscopy image?

Answers

Speaker Bio

Loic Royer

Loic Royer

Dr. Loic Royer is a Group Leader at the Chan Zuckerberg Biohub. Royer initially studied engineering and then obtained a master’s degree in artificial intelligence followed by a doctoral degree in bioinformatics from the Dresden University of Technology in Germany. He continued his scientific training as a postdoc in the laboratory of Dr. Gene Myers,… Continue Reading

Leave a Reply Cancel reply