Videos that introduce new weapons often include slick CG scenes. A beam of light slices across the sky and the target neatly disappears. It looks convincing. But what does that scene prove?
A picture, a calculation and a test each show something different. This article measures that difference with a ruler used by NASA and the U.S. Government Accountability Office (GAO): technology readiness. It pairs well with Why lasers are studied by calculation first.
Pictures, calculations and tests are different evidence
First, some terms. CG or animation puts a scene someone imagined onto a screen. It is good for showing an idea, but the light and motion in it behave exactly as the maker decided.
Simulation is different. It feeds values into a model, a set of equations built from physical laws and material properties, and calculates the result. A calculation can answer what the model contains, but not what the model leaves out.
A test puts real hardware in conditions close to real ones and measures what happens. And a test range is still not the place where the system will be used. So evidence stacks up like a staircase.
Think of cooking: a photo of a dish cannot show its taste. Working through the recipe only lets you guess at it. You know the taste only after you cook and eat it. Real weapon development goes through far more stages, and at each one it records what was tested and where.
A ruler for maturity: readiness levels 1 to 9
The ruler for those stages is the technology readiness level (TRL). NASA describes TRL as a measurement system for judging how mature a given technology is. There are nine levels, with TRL 1 the lowest and TRL 9 the highest.
GAO's guide to technology readiness assessments (GAO-20-48G) states the root of this ruler in one sentence. TRLs describe technical maturity rising step by step, judged on capabilities that have been demonstrated, that is, tested. Broadly, level 1 is a written study of the basic idea, and level 4 or so is a demonstration in the lab. Level 9 is a technology that has been tested, proven, built into a product and run successfully in its intended environment.
The DoD hardware table that GAO presents makes the middle levels clearer.
| Level | Gist of the DoD table |
|---|---|
| TRL 5 | Basic parts are joined with fairly realistic supporting elements and tested in a simulated environment |
| TRL 6 | A prototype is shown in a relevant environment. Examples: tests in a high-fidelity lab or a simulated operational environment |
| TRL 7 | A real system prototype is shown in an operational environment such as an aircraft, a vehicle or space |
| TRL 8 | The system is finished in its final form and qualified by test. Example: developmental test and evaluation (DT&E) |
| TRL 9 | The final form is used under mission conditions. Example: conditions met in operational test and evaluation (OT&E) |
Where simulation fits
Does that make calculation useless? Not at all. NASA notes that at TRL 5, simulations need to run in settings as close to real as possible. Calculation also helps prepare tests and interpret their results.
GAO also writes that testing and performance checks may be backed up by calibrated models and simulations. Two words matter here. "Calibrated" means the model has been tuned against real measurements, and "backed up" means it helps alongside testing rather than taking its place.
The same holds for lasers. Why lasers are studied by calculation first covers checking a model against measurements. How a beam spreads is worked out in The basic idea of beam propagation. Tools anyone can use are in A look at openly available calculation tools.
A test environment is not the real one
Even a test is not the end. Under the GAO guide, a key technology should in the end prove its performance in the operational environment. Only then does it count as mature enough to build into a system, and that is usually TRL 7. A few special cases exist, space among them, where a test in the operational environment is not practical, and there TRL 6 can be acceptable.
So test results should come with a note on where the test setting departs from real use. The note needs enough detail for assessors to see that difference. One good run is not enough either. The same result has to repeat often enough to be statistically significant.
Words need care too. Terms like "simulated environment," "relevant environment" and "operational environment" often mean different things in different organizations and fields. In the guide, a relevant environment is set separately for each key technology. It draws on the parts of the operational environment that pose a risk to that technology working well.
So it is not a full copy of the real world but a selection of the conditions that threaten that technology. The high-fidelity lab and simulated operational environment in the TRL 6 row are examples of places for such tests. The mindset of the assessors matters as well. The guide also warns that optimism, common in acquisition programs, can color assessment findings or how they are read.
Stages seen in public cases
How do real programs climb these steps? Here are three cases from other articles in this series.
DARPA's OFFSET program on drone swarms is structured around two parts side by side. One is a virtual world, networked and running in real time, that follows physical laws. The other is frequent live experiments with a range of uncrewed air and ground platforms.
At the time of the U.S. Army article, test items from four companies in the LASSO loitering munition program were going through risk reduction testing at Yuma Proving Ground. Before explosives are added, safety flights confirm the safety features. The items then pass through conditioning chambers, rough handling and climate tests, ending with flight tests using high explosives to check reliability and durability. The data gathered feeds decisions on fielding.
The U.S. Department of Defense announced that its October 2016 test of 103 Perdix drones confirmed the reliability of a design made only of commercial parts under possible deployment conditions. With that announcement too, you can ask: under what conditions, how many times, and at which stage of testing?
Questions to ask of an announcement
When you watch a weapon announcement or promotional video, these five questions help you weigh the evidence.
- Is what you see a picture, a calculation or test footage?
- If it is a calculation, was the model checked against real measurements?
- If it is a test, was it in a lab, a simulated environment or the operational environment?
- Was it done once, or repeated with the same result?
- Who is speaking: the developer, or a separate evaluator?
These questions are not meant to belittle pictures or calculations. Knowing that each stage supports a different kind of claim lets you read how far one sentence in an announcement really goes.
Reading on with the laser articles
Laser weapon stories often mix phrases like "successful test," "fielded" and "deployed." How to read laser weapon news traces which development stage each one points to, using the order of official announcements for Korea's laser anti-aircraft weapon.
Read that article with the readiness ruler from this one beside it, and you can tell which step of the staircase a given "success" comes from.
What to read next
- Why lasers are studied by calculation first: what calculation does well and badly, and how it pairs with measurement.
- How to read laser weapon news: reading "kW-class," "range" and "successful test" together with their conditions.
- A look at openly available calculation tools: a tour of public tools for calculating light and lasers.