Engineering the flow: SDSU models liquid argon inside DUNE

Mechanical engineers need to master fluid dynamics because gases and liquids are in motion all around us. Fluids are involved in everything from the airflow that makes up weather patterns to the flow of gasoline vapor inside a piston-driven engine, to the way groundwater flows through aquifers and into our drinking water wells. The effort to model and understand how fluids work in different systems can involve challenging concepts, mind-bending mathematics, and complex algorithms that can only be solved by high-end computers.

For nearly a decade, two researchers at South Dakota State University have been working to understand the intricate details of the flow of liquid argon inside giant tanks that are designed to detect elusive particles called neutrinos.

The Long Baseline Neutrino Facility and the Deep Underground Neutrino Experiment (LBNF/DUNE) are being built at the Fermi National Accelerator Laboratory in Illinois and the Sanford Underground Research Facility (SURF) in the Black Hills of South Dakota. LBNF/DUNE will improve precision understanding of neutrinos and potentially unlock some critical mysteries of the universe.

Make no mistake, the LBNF/DUNE is a colossal undertaking. The DUNE collaboration comprises more than 1,400 scientists and engineers from over 200 institutions in 38 countries plus CERN in Switzerland. The detectors for DUNE, located inside massive caverns that have been excavated nearly a mile underground at SURF, include giant cryogenic tanks, each about five stories high, that will be filled with liquid argon, which must be kept at about 300 degrees below zero Fahrenheit.  (This short video helps explain DUNE).

One of the many engineering challenges of DUNE is mapping out slight differences in  temperature and purity within this massive volume of liquid argon at a very high-level of precision. The entire volume is monitored continually and temperature differences on the order of 20-thousandths of one degree Kelvin throughout the massive tank containing  17,500 tons of liquid argon must be understood to properly reconstruct neutrino interactions in the detector. On top of this, the liquid argon must be kept extremely pure as a single drop of water in this building sized tank of liquid argon could prevent the interactions of interest from being detected.

Dr. Greg Michna, an associate professor of mechanical engineering at South Dakota State University (SDSU), is one of the researchers working to model the flow of liquid argon inside DUNE. He says predicting these ultra-small temperature differences  seems daunting at first, but it is achievable.

“Our goal is to create a three-dimensional map and predict the purity and temperature within the detector, so that the people who are doing the calibration for the physics experiments will know areas of relative purity, and maybe relative impurity and the temperature variability,” said Michna.

The temperature and purity across such a large volume of liquid argon can be influenced by many things: some heat comes in through the insulated walls of the cryogenic tanks and this causes the locally warmed liquid to rise; the fluid is also moving from the tank to the cryogenic coolers and back; the components inside DUNE, like the anode plane assemblies can impact flow and temperature, the numerous plastic covered wires and pipes—can all emit tiny amounts of impurities and change the experimental readings.

“Now, to get to the level of fidelity and detail that we're looking for in this system, that's where we need the high-performance computing capability,” said Dr. Stephen Gent, a professor of mechanical engineering at SDSU and an expert in computational fluid dynamics. Gent works alongside Michna on modeling the flow of argon inside DUNE.

The modeling team employs High Performance Computing (HPC) clusters that are part of the Research Computing Group at SDSU. The computers that model liquid argon flow inside these systems are so complex, it takes these large computers about five days to run one model of the current simulation. Gent says this is a huge improvement over their computing capacity when they started in 2015 when these same computations would have taken months to complete.

“We're actually on our third HPC cluster since we've started this project,” Gent says. “SDSU has increased this computing capacity by about 40 times since the start of the project. These computer updates are thanks to state resources, along with funding from the National Science Foundation, and federal grants. We have to give thanks to the Research Computing Group here at SDSU and all our funding partners who make this work possible.”

The process involves building the computer models and testing them against the real world in the prototype DUNE detectors called ProtoDUNE, located at CERN and Fermilab liquid-argon experiments including MicroBooNE, ICARUS, SBND, DUNE 2x2.  The data from the real-world tests both inform new computer models and the eventual operation of the experiment itself.

“Validation of our results has been a large focus for the past couple of years just to make sure that our simulations can be trusted and that our model is accurately predicting what is happening inside the ProtoDUNE detectors,” says Michna.

Michna and Gent say they learned a great deal in the first version ProtoDUNE and they are excited to see the data that will come off the latest version of ProtoDUNE that began the liquid argon filling process this spring at CERN. ProtoDUNE contains many more liquid argon temperature and purity sensors.

“The idea is we can use the same modeling approaches to be able to predict what will happen in the DUNE detectors that are going into SURF before they're actually commissioned there,” Gent said.

Both Gent and Michna stress that working with the international collaboration on this world-class experiment is a unique opportunity not only for them as academics, but more so, for their students. Michna says this project has included seven graduate students who landed the rare opportunity to contribute to an experiment of this caliber.

“The huge payoff for me is seeing the students grow,” Michna said. “They're working on these projects, and they graduate and go and work in South Dakota, or in this region, and take the skills they're learning out to industry, that’s pretty great to see.”

The work by students and some 1,400 established researchers on DUNE may someday contribute to the fundamental understanding of our universe. It’s possible that the outcome of this work could someday lead to a Nobel Prize. If this occurs, it won’t just be a small number of top scientists who have earned that Nobel, it will be thanks to every individual who contributed to the success of the collaboration over the decades.

“There are very few things in the world of research, especially if they're substantial, that can be done by one or two people. These grand scientific and engineering challenges, require teams of experts who are very committed and who work for a long, long time,” says Gent. “Knowing that Steve and I and our students might help make a Nobel happen someday is exciting. It’s part of what makes this work so rewarding,” Michna adds.